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Detailed Description Text - DETX (5) : 
A receiver for a hand-held remote 
control unit may also be connected to the 
serial interface 14 so that the user may be 
provided with a hand-held infrared , 
RF or other type of remote 22 for 
commanding the central processor to perform 
the various available tasks. 



Detailed Description Text - DETX (9) : 
The security system 38 used in the 
automated environment, may preferably be 
a Silent Knight Model 4000 security system, 
available from Silent Knight, of 
Minneapolis, Minn. Such a home security 
system 38 is connected to an 
RS-232 - to- security system translator 36 
which is in turn connected to the 
central processor 10 by means of the serial 
interface 14 and a data bus 12. It 
can provide simultaneous RS-232 
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communications with eight separate external 
systems. When thus configured, the seria 
interface 14 can communicate 
directly with any external system, such as 
an Biographies touchscreen 
controller, or any other system that uses_ 
standard RS-232 protocols. If, as is 
the case in the preferred embodiment, the 
external device uses a nonstandard 
protocol, then an RS-232 protocol 
converter, such as element 36, can be used 
to 

convert between the RS-232 protocol and the 
protocol utilized by the device to 
be controlled. That same interface, as is 
shown as element 14 in FIG. 1, may 
also be used to connect the central 
processor 10 and its data bus 12 through a 
protocol converter 34, if necessary, to 
various home automation buses such as 
CEBUS and SMART HOUSE. 



Detailed Description Text - DETX (23) : 

The present home automation system is 
also hardware compatible with many 
other readily available graphics boards 
which offer extended capabilities. For 
example, by incorporating a different 
graphics card and making minor 
modifications to the home automation system 
software, the present system can 
provide graphics up to 1024 . times . 768 
pixels. Another readily available card 
offers compatibility with the standard NTSC 
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video format used in United States 
television sets. The system may also 
include high performance graphics display 
processor cards for advanced graphics 
effects including animation. 

Detailed Description Text - DETX (24) : 
Using the high resolution graphics 
discussed above in concert with high 
resolution touchscreens, many functions in 
a home or other environment can be 
controlled. For example, audio/video 
entertainment systems can be actuated and 
caused to perform all of the functions 
available by use of their respective 
remote controls. This feature is 
accomplished by means of an infrared 
emitter 

device which is utilized, under control of 

the central processor 10, to input 

to the audio/video device being controlled 

the appropriate code for controlling 

the desired function. Alternatively, a 

direct serial link or other electronic 

interface could also be used to control the 

audio/video device. For example, 

the present home automation system can, 

utilizing a touchscreen, energize a 

connected standard television receiver and 

control the volume, select the 

channel, select the stereo or mono mode, 

and, as described, control any 

function which can be controlled using that 

television receiver's remote 
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control . 



Detailed Description Text - DETX (48) : 

Functions that can be scheduled in this 
manner include turning a light or 
appliance on or off, watering the lawn, 
turning on different lighting moods at 
different times of the day, turning on 
audio music or a television station, VCR 
recording of television programs, operating 
modes or features of the home 
automation system itself, operating modes 
of a security system, electric locks, 
reminders for the system user, operating 
modes of a telephone system and 
automatic information retrieval. As will 
be described further herein, the same 
types of displays may be used directly to 
control entertainment systems by 
working with a graphics representation of 
the entertainment system and its 
distributed speakers and sources. The user 
controls the system by selecting 
his audio or video source from a graphics 
display and then selecting the rooms 
in which he would like the audio and video 
played. 



Detailed Description Text - DETX (53) : 

The audio/video touchscreen sub-menu 
shown in FIG. 3b presents the user with 
the various functions which can be 
controlled by the present home automation 
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system as disclosed herein. The system can 
control a VCR, a television, select 
certain AM or FM presets on a receiver, 
control a compact disk, distribute 
audio throughout the home, be used to 
operate the volume, tone and other 
controls associated with the audio and 
video equipment controlled from the 
audio/video touchscreen, and turn the 
audio/video system off. Obviously, 
additional similar pieces of equipment and 
functions may likewise be 
controlled. 



Detailed Description Text - DETX (56) : 
FIG. 3e shows yet another sub-menu 
touchscreen which can be accessed from 
the audio/video sub-menu touchscreen of 
FIG. 3b. The touchscreen shown in FIG. 
3e allows the user to select an audio 
source which may then be distributed, 
selectively, throughout the home through 
high fidelity speakers placed 
strategically in different zones throughout 
the home. The selections include 
the output from the compact disk, from a 
stereo receiver, from the VCR, from a 
television and also a function to turn the 
distributed audio off. 



Detailed Description Text - DETX (85) : 

The present home automation system, by 
reason of utilizing an AT compatible 
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bus, allows the use of a wide variety of 

video options. For example, by use of 

certain display driver cards, a range of 

video resolutions from high resolution 

displays for use with detailed floor plans 

to lower resolution menus can be 

shown. That provides for faster loading of 

the information into the central 

processor and out to the monitor 56. Also, 

by using a monitor that can show 

both standard video as well as the high 

resolution display graphics, the 

display screens 5 6 can double as 

televisions or security monitors. 



Detailed Description Text - DETX (86) : 

This function is accomplished by the use 
of a multi-resolution video display 
driver card such as the VGA VIP card or the 
VGA Wonder card, available from a 
ATI. It also uses as the primary video 
monitor, the Mitsubishi AUM-1381A which 
can display either computer-generated 
EGA/VGA video or standard NTSC television 
video . 



Detailed Description Text - DETX (91) : 

A preferred external security system to 
which the present home automation 
system may be interfaced is the Silent 
Knight 4000 security system, available 
from Silent Knight. That home security 
system is shown as element 38 in FIG. 



09/02/2003, EAST Version: 1.04.0000 



1. It is interfaced to the central 
processor 10 by means of a custom 
RS-232 -to-security system translator 36, 
which in turn is connected to the 
serial interface 34a and then through the 
AT bus 12 to the central processor 
10. The interface or translator 36 is a 
microprocessor-based translator and 
controller . 



Detailed Description Text - DETX (93) : 

As shown in FIG. 2, a four wire bus is 
utilized by the Silent Knight home 
security system 38, two wires for power and 
the other two for data 

transmission. The interface translator 36 
connects to all four wires of that 
bus . Data originating in the security 
system 38 is first level shifted from a 
data stream between +12 volts and ground to 
a data stream between +12 volts and 
-12 volts and then passed to an input bus 
of a Z80-based microcomputer. Under 
control of an EPROM resident program, the 
Z80 microprocessor translates data 
and instructions appearing on the bus. 
Such data and instructions exit the 
translator 36 in RS-232 format, from which 
they ultimately reach the central 
processor 10. When the home automation 
system gives a command to the security 
system, it passes through the Z80 
microprocessor, is converted to the proper 
format and timing for the security bus. 
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passes through another level shifter 

and is then converted to the +12 and ground 

format used by the security system 

38. 



Detailed Description Text - DETX (102) : 

A flow chart of the functions performed 
by the Subsystem Interrupt Handler 
is shown in FIG. 4b. All of the tasks and 
devices based on a serial line, such 
as the touchscreen, hand-held remote 
control unit and security system, are 
supported by the interrupt handler, running 
in the background. When a byte of 
data is received at any serial port, an 
interrupt request line is raised, which 
causes the interrupt handler to execute. 
The interrupt handler determines 
which serial port caused the interrupt, 
copies the new data into that port's 
queue and then increments that queue's 
pointer. The Polling Loop, which is 
shown in FIGS. 5a and 5b, monitors the 
queue pointer and jumps to the queue's 
task when the queue pointer moves. 



Detailed Description Text - DETX (103) : 

As shown in FIG. 4b, when the interrupt 
request line is raised the interrupt 
handler begins to execute. Initially, the 
interrupt handler jumps to the 
interrupt handler routine 300. The 
interrupt handler routine then determines. 
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by asking each serial chip, which serial 
line generated the interrupt 302 which 
caused the interrupt request line t be 
raised. Upon determining which serial 
line generated the interrupt, the 
appropriate data byte is pulled from the 
identified serial lines, RX register 304 
and is placed in the appropriate 
serial queue 306. That queue's pointer is 
then incremented 308. If the queue 
pointer is greater than the queue size, 
then the queue pointer is wrapped 
around. The interrupt handler routine then 
ends and returns to the normal 
program 310. 



Detailed Description Text - DETX (149) : 

When a Task receives data, it must first 
evaluate it. The data validation 
routine (DVR) consists of a sequence of 
states, in which each state evaluates 
one byte of data . If the byte is rejected, 
the state resets the DVR. If the 
byte is not rejected, the state increments 
the state variable to the next DVR 
state and returns to the Polling Loop, thus 
allowing the next byte of data to 
cause the next DVR state to execute. The 
communication protocol for that 
task's device is encoded into the DVR, in 
order to form the criteria by which 
the DVR will accept or reject data. The 
DVR also simultaneously decodes the 
data to provide necessary information to 
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1 



the remainder of the task. 



Detailed Description Text - DETX (220) : 

In the event that there is more 
Security/Fire Interface data available at 
step 826, or in the event that at the 
V-state determination 802, the current 
state is V-state 3, a determination is made 
as to whether a valid command byte 
is present 828. If no valid command byte 
is present, the V-state is reset 830. 
If a valid command byte is present, a 
determination is made at step 832 as to 
whether the length is greater than 1. If 
the outcome of that determination is 
affirmative, then the V-state is set at 
step 834 to V-state 4. If the 
determination of the length at step 832 is 
such that the length is less than 1, 
then the V-state at step 836 is set to 
V-state 5. After the V-state is set to 
either V-state 4 or V-state 5 at steps 834 
or 836, respectively, a 

determination is again made as to whether 
more Security/Fire Interface data is 
available 838. If no more Security/Fire 
Interface data is available at step 
838, then the program returns to the top of 
the Polling Loop 500. 



Detailed Description Text - DETX (221) : 

If more Security/Fire Interface data is 
available at step 838 or if the 
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current V-state at step 802 is determined 

to be V-state 4, then the data byte 

is stored 840 and a determination is made 

as to whether all data has been 

received from the Security/Fire Interface 

842. If all data has been received 

from the Security/Fire Interface, then the 

V-state is incremented by one at 

step 844 and a decision as to whether more 

Security/Fire Interface data is 

pending is made at step 846. If it is 

determined at step 842 that all data has 

not been received, then the Validation 

subroutine moves directly to step 84 6 to 

determine whether more Security/Fire 

Interface data is available. If no more 

security/fire data is available at step 

84 6, then the program returns to the 

top of the Polling Loop 500. 



Detailed Description Text - DETX (222) : 

If the answer to the query at step 846 
is that more Security/Fire Interface 
data is available, then the program goes to 
step 848, stores the nth data byte 
at step 840 and then goes to V-state 5. 
Obviously, as set forth in FIG. 8a, if 
at step 802 the current state is V-state 5, 
then the program would branch 
directly to V-state 5 at 850. 



Detailed Description Text - DETX (223) : 
As shown in FIG. 8b, from V-state 5 at 
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step 850, the program then removes 

the "complete security message" from. the 

ISIS 852 and sets the V-state back to 

0 at 854. A determination is then made as 

to whether the checksum byte is 

valid 856. If the check siim byte is not 

valid, then the program returns to the 

top of the Polling Loop 500. If the 

check s\im byte is determined to be valid at 

step 856, then the program moves into the 

immediate response portion of the 

Security/Fire Interface subroutine and 

acknowledges the message received to the 

Security/Fire Interface 858. A 

determination is then made of the current 

state 

of the Security/Fire Interface, whether it 

is in the Neutral State, the "disarm 

system" or other Contextual States, at step 

860. If it is in the Neutral 

State, the program proceeds to determine 

whether the current command is an 

alarm at 862. All commands that are not 

alarms are ignored and the program 

returns to the top of the Polling Loop 500. 

Detailed Description Text - DETX (256) : 

FIG. 15a shows a flow chart of the 
Validation routine of the Input/Output 
Task. Upon determining that a new data 
byte is present in the input/output 
queue 1500, a determination is made of the 
current V-state 1502 to which the 
program then jumps. If the current V-state 
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at 1502 is V-state 0, then a 
determination is made as to whether the new 
data byte in the input/output queue 
is a valid conimand byte 1504. If the 
current byte is a valid command byte^ the 
command is stored 1506 and then a 
determination is made as to whether the 
stored command is an acknowledgment 1508. 
If the stored command is an 
acknowledgment, then the value of 1 is 
stored to the variable length 1510 and 
then the system jumps to the Immediate 
Response routines 1512. 

Detailed Description Text - DETX (257) : 

If the current byte is not a valid 
command byte at step 1504, the system 
then returns to the top of the Polling Loop 
500. 



Detailed Description Text - DETX (258) : 

If the command stored at step 1506 is 
not determined at step 1508 to be an 
acknowledgment, then the value of 4 is 
stored as the variable length at step 
1514 and the "complete 10 controller 
message" ISIS event is inserted 1516. A 
determination is then made as to whether 
any more input/output data is 
available 1518. If no more input/output 
data is available, then the system 
returns to the top of the Polling Loop 500. 
If more input/output data is 
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available at step 1518 or if the V-state at 
step 1502 is determined to be 
V-state 1, then the new data byte in the 
output queue is stored as data 1520 
and then a determination is made as to 
whether all data has been received 1522. 
If all data has not been received, then the 
system returns to the top of the 
Polling Loop 500. If all data has been 
received at step 1522, then the V-state 
is reset 1524 and the "complete I/O 
controller message" ISIS event is removed 
at step 1526. The system then jumps to the 
Immediate Response routines 1512. 

Detailed Description Text - DETX (261) : 

If the command at step 1528 is a digital 
status command, then the system 
decodes the port number, decodes the data 
bytes corresponding to the input 
board 1536 and then goes to determine the 
current state at 1538. Likewise, 
after the new and old bit masks are 
compared and a determination as to which 
inputs caused the transition report is made 
at step 1534, a determination is 
then made of the current state 1538. If 
the current state at step 1538 is the 
Neutral State, then a determination is made 
as to whether the current command 
is a transition report 1540. If the 
current command is not a transition 
report, then all other commands are ignored 
and the system returns to the top 
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of the Polling Loop 500. 

Detailed Description Text - DETX (306) : 

As shown in FIG. 12a, the main menu of a 
touchscreen control-based system 
similar to that disclosed herein in 
connection with the expandable home 
automation system is shown. Equipment such 
as audio equipment, TV tuners, 
VCRs, lights and drapes, overhead 
projection and slide presentation equipment 
can be controlled merely by touching the 
appropriate function box on the 
touchscreen menu shown in FIG. 12a. Other 
functions such as quitting, muting 
the sound, volume up and down, and lights 
on and off may also be provided. 

Detailed Description Text - DETX (308) : 
FIG. 12c shows the TV tuner sub-menu 
touchscreen which is selected by 
touching the TV tuner functional block of 
FIG. 12a. The TV tuner sub-menu 
touchscreen allows the user to turn the TV 
on by selecting a particular 
channel, to turn the TV off, and to scan 
the channels using up and down 
function blocks. 



Detailed Description Paragraph Table - DETL 
(1) : 

TABLE 1 
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Example of a Specific Task 

Description - Task 1: the Fire/Security 

Interface 

Data 

Validation: The data validation 
consists of the following states: 1st 
Synch byte: Must be equal to. 52 or data 
is rejected 2nd Synch byte: Must be equal 
to 126 or transmission is rejected 
Length byte: Receive length of message 
Command byte : Receive command byte 
Data bytes : Receive appropriate number of 
data bytes ^ if any Checksum byte: 
Must be equal to sum of all bytes in 
message MOD 256 or transmission is 
rejected Immediate Response: Call 
security device driver to send 
acknowledgment of receipt of message. 
Neutral State: If command is Fire 
Alarm, then set Master Touchscreen task to 
Fire Alarm State, jump to Fire 
Alarm State in Master Touchscreen Task, 
end if If command is Security Alarm, 
then set Master Touchscreen task to 
Security Intrusion State, jump to 
Security Intrusion State in Master 
Touchscreen Task end if If command is any 
other type of alarm, then call lighting 
moods device driver to turn on all 
lights end if ignore all other commands 
return to Polling Loop Contextual 
States: Waiting for response to disarm 
system request: if command is 
acknowledgment then result = good jump to 
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Disarm Security System substate of 
Master Touchscreen Task else if command 
is system already disarmed result = 
already disarmed jump to Disarm Security 
System substate of Master 
Touchscreen Task else if command is 
password rejected result = password 
rejected jump to Disarm Security System 
substate of Master Touchscreen Task 
else execute Neutral State without 
changing Contextual State end if Waiting 
for a response to shunt zone command: if 
command is acknowledgment then 
result = good jump to Shunt Zone substate 
of Master Touchscreen Task else if 
command is rejected password then result = 
rejected password jump to Shunt 
Zone substate of Master Touchscreen Task 
else execute Neutral State without 
changing Contextual State end if Waiting 
for interface to return zone map if 
command is zone map then result = good 
jump to substate that requested a zone 
map else if command is rejected password 
result = rejected password jump to 
substate that requested a zone map else 
execute Neutral State without 
changing Contextual State end if Waiting 
for response to arm system request: 
if command is acknowledgment then result = 
good jump to Arm Security System 
substate of Master Touchscreen Task else 
if command is system already armed 
result = already armed jump to Arm 
Security System substate of Master 
Touchscreen Task else if command is system 
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not ready result = not ready jump 
to Arm Security System substate of Master 
Touchscreen Task else if command is 
password rejected result = password 
rejected jump to Arm Security System 
substate of Master Touchscreen Task else 
execute Neutral State without 
changing Contextual State end if 



Detailed Description Paragraph Table - DETL 
(4) : 

TABLE 4 



Action Routines & Device 
Drivers 



Musicon: Turn on distributed 
audio system to a set source to a set 
group of zones. Musicoff: Turn off 
distributed audio system. 

HVACzone#setbackon: Sets a Heating/Cooling 
zone to 

an energy saving temperature. 
HVACzone#setbackof f : Sets a 
Heating/Cooling 

zone to a normal temperature. Unlockdoor: 
Unlocks a given door. Lockdoor: 
Locks a given door Bathtempveryhot : Sets 
bath temperature to equivalent of 
very hot. Bathtemphot: Sets bath 
temperature to equivalent of hot. 
Bathtempwarm: Sets bath temperature to 
equivalent of warm. Fillthebath: Fills 
the bath to a preset level and 
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temperature. Emptythebath : Empties the 
bath . 

Setvacmode: Turns on vacation mode for the 
home. Resetvacmode : Turns off 
vacation mode for the home . 
Randomlightson : Turns on random lighting. 
Randomlightsof f : Turns off random 
lighting. Ltmoods : Sets a certain lighting 
mood according to the value passed it. 
Speakerson: Turns on speech response 
speakers according to the values passed 
it. Speakersoff: Turns off speech 
response speakers according to the values 
passed it. AMX: Formats and 
transmits a message to the Audio/Video 
Controller (IR Emitter) . Sentoioc: 
Formats and transmits a message to the 10 
Controller. Sendtosec: Formats and 
transmits a message to the S/F Interface. 



Detailed Description Paragraph Table - DETL 
(5) : 

TABLE 5 



Support Environment 
Functions : 



Display Cursor: Handles 
cursor movement and touchbox dynamics . 
Pullbyte: Pulls a byte of . data from a 
specified Serial Input Queue. 
Colorchange : Changes any color within a 
specified rec- tangular area to any other 
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color, according to the values 

passed it. Instimeevent : Inserts an event 

into the ISIS queue, according to 

the event type and event time passed it. 

Deltimeevent : Deletes an event from 

the ISIS queue, according to the event 

time passed it. Checktouch: Compares 

recognition templates and touch 

coordinates with the locations of the 

current 

touchboxes on the screen, and returns the 
number of any activated touchbox. 
Reloadsched: Reloads the daily schedule 
from a data file, screening out 
events that are not to occur on that day. 
Playafile: Speak a phrase. 
Playmemsequence : Speak a phrase followed 
by individual words, such as 
numbers, i.e. "The temp- erature is", 
"90", "5", "degrees". Loadscreen: Load 
a new menu screen and reset various 
graphics status variables. Loadslice: 
Load a portion of a screen. Loadvocab: 
Load a voice recognition vocabulary 
into a speech board. Initrecognition: 
Initialize voice recognition for a 
menu. Spchutil: A variety of routines 
designed to manage the speech boards in 
a cohesive, non-inter- fering manner. 
Commbyport: A general serial 
transmission routine. Avrsof terror : Error 
handler for code-detected error 
conditions. Codeerror: Error handler for 
software run-time errors. Readana: 
Reads an analog voltage and/or temp- 
erature from an analog input. Readdigio: 
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Reads the current values of all the inter- 
rupting digital inputs. 

Metrabyteread: Reads the current values of 
a group of passive digital inputs 
according to the value passed it. 
Metrabyteon: Turns on a digital output 
according to the values passed it. 
Metrabyteof f : Turns off a digital output 
according to the values passed it. 
Metrapulseon : Pulses a digital output on 
for the duration passed it. 
Metrapulseof f : Pulses a digital output off 
for 

the duration passed it. 
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Detailed Description Text - DETX (4) : 

In addition to controlling the instant 
automation system by means of one or 
more touchscreens 16a and 16b, the central 
processor may also be connected to a 
standard IBM AT or compatible computer 
keyboard 18. While such a keyboard may 
be utilized by the user, it will more 
likely be utilized by technical personnel 
during set up and maintenance procedures. 
A standard PC computer keyboard may 
also be effectively utilized. 

Detailed Description Text - DETX (43) : 

It should also be remembered that an 
important aspect of the present 
invention is the commonality of use between 
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input devices. For example, both 
the computer keyboard 18 and the voice 
recognition system use the touchscreen 
displays for cues and visual feedback. 
Wall-mounted telephone touch-tone 
buttons can be used to control a subset of 
the touchscreen commands. In 
addition, although not shown specifically 
in FIG. 1, other user interfaces may 
be added to the home automation system, 
such as track balls, mice, joy sticks, 
light pens and other cursor control 
devices. In addition, while a hand-held 
remote control unit is presently used, . 
wireless versions of any of the present 
interfaces shown in FIG. 1, such as 
wireless portable touchscreens and wireless 
microphones or radios for remote voice 
recognition, may also be used. Voice 
recognition may also be used over the 
telephone when that technology becomes 
available . 



Detailed Description Text - DETX (44) : 

In order to implement the use of 
multiple types of user devices in a modular 
manner which allow for different types of 
devices in different parts of the 
home to be used to control the home 
automation system described herein, the 
following equipment may be simultaneously 
connected to the home automation 
system central processor: (1) a standard PC 
computer keyboard 18; (2) plurality 
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of dry contact switches 28a and 28b may be 

connected to the central processor 

10 by means of a solid state input module 

and a pull up resistor to a Metrabyte 

MSSR-32 solid state relay board which is 

shown as process controller 26b, which 

is in turn connected to the parallel 

interface 24b, which may be an MDB-64 

parallel controller connected to the AT bus 

12 of the central processor 10; (3) 

multiple Elographics Accutouch touchscreens 

16a and 16b connected serially to 

the central processor 10 by means of the 

Digicom COM/8 multiport serial 

interface 14; (4) a plurality of hand-held 

remote receivers 20 may be serially 

connected to the central processor 10 by 

means of the serial interface 14; (5) 

a telephone system 62 with a plurality of 

hand sets, or a single line standard 

telephone system may be connected to the 

central processor 10 through a 

telephone interface 60 and the speech 

processor 58 and the AT bus 12; (6) 

multiple voice recognition locations may be 

connected to the central processor 

10 through Crown PZM microphones or other 

remote microphone systems' 64 which 

are in turn connected to respective 

TISPEECH speech processors 58 running 

standard voice recognition software which 

themselves are connected by the AT 

bus 12 to the central processor 10; and (7) 

multiple voice response locations 

with remote speakers 66 which provide 

spoken information and instructional cues 
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to the user. All of the foregoing 
equipment is simultaneously connected to 
the 

central processor 10 and may be used to 
perform a plurality of functions. 



Detailed Description Text - DETX (81) : 

The foregoing types of information can 
also be requested with any wired or 
wireless cursor control or general input 
device, such as a mouse, touchscreen 
16, light pen, joy stick, keyboard 18 or 
voice recognition system. 

Detailed Description Text - DETX (115) : 
5 . Keyboard 



Detailed Description Text - DETX (126) : 

If there is no input/output controller 
data at step 516, a determination is 
then made as to whether a key has been 
pressed 522 on the keyboard 524. If a 
key has been pressed, then certain 
diagnostic routines 52 6 may be called and 
then the program returns to the top of the 
Polling Loop 500. Alternatively, 
although not shown in FIG. 5a, once a 
determination that a key has been pressed 
522 is made, another determination as to 
whether a diagnostic routine has been 
called may be made. If a diagnostic 
routine has not been called, then the 
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system will interpret the keyboard input to 
be instructions or data and will 
execute those instructions or data 
accordingly. 



Detailed Description Text - DETX (131) : 
FIG. 5c is a flow chart showing the 
voice recognition portion of the Polling 
Loop. After it has been determined that a 
key of the keyboard 524 has not been 
pressed, a determination is made as to 
whether the voice recognition circuitry 
is enabled 554. If that determination is 
affirmative, then a determination is 
made as to whether a telephone call is 
currently active 556. If a telephone 
call is currently active, then the system 
skips to the touchscreen 1 queue at 
step 558. 



Claims Text - CLTX (19) : 

a plurality of additional communication 
interface means, including at least 
two of a keypad, serial data keypad, a 
voice recognition system, hand-held 
remote control, computer keyboard and 
telephone, said at least two additional 
communication interface means being 
simultaneously connected with each other 
and said visual communication interface 
means to said programmed data processor 
such that said user may communicate 
commands to said system using any of said 
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connected communication interfaces. 
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[57] ABSTRACT 

A system for and a method.of providing an expandable 
home automation controller is disclosed which supports 
multiple numbers and multiple different types of data 
communications with both appliances and subsystems 
within the home as well as systems external to the 
home. The system is based upon a central processor, 
such as a microprocessor-based computer, and is con- 
nected by means of a data bus to control the various 
products and subsystems within a home or commercial 
building, such as lighting systems, security systems, 
various sensors, multiple external terminals, as well as 
to allow for the input of commands by a variety of 
means such as touchscreens, voice recognition systems, 
telephones, custom switches or any device capable of 
providing an input to a computer system. The system 
functions can be readily controlled by the user utUizing 
a high resolution graphics display and associated touch- 
screen interface. 
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signed as a master controller for intelligent subsystems. 

EXPANDABLE HOME AUTOMATION SYSTEM • Although it can also control simple devices directly, the 

primary method of control of the present invention is by 

BACKGROUND OF THE INVENTION means of multiple communication channels which are 

-T^ . • ^««^«i,t«r r*«r,trrtiiAH ^ compatiblc with or convertible to a wide variety of 
The Dresenl mvention relates to computer controJiea \ , ^ • i -n. , 
syst^s l^automaling a plurality of diverse electri- standard data commumcauons protocols. Thu . the 
Sr<^nU0llable subsystti^s. More particularly, the system disclosed m this apphcanon can b^^^^^ 
^nTinvention relate to a microVrocessor-based virtuaUy any type of electncally comroUed device that 
dectronic control system for use in the automation and may be presently found m a home or can conceivably be 
control of electrical subsystems found in the home envi- comiected to either the standard dau cominunication 
ronment. Such systems may also be utilized in commer- buses in existence today, local an^ networks or future 
cial applications ^o™^ automation data buses such as CEBUS or tne 
In the past, sophisticated electronic control over SMART HOUSE bus*. The system may also be con- 
complex systems has been limited primarily to industrial nected to devices to be controlled directly by parallel 
applications, and occasionally commercial applications. and serial ports- 
Such electronic control systems usually required exten- With the innovative expansion capabilities of the 
sive and costly equipment, as well as technically trained mventive system, simultaneous operation of multiple 
operators. However, as consumer products and systems types of user devices can now be achieved. For exam- 
become more and more complex, untrained people in pie, the home automation system described herein may 
home environments have been increasingly desirous of ^ be connected to simple keyboards, serial data keypads, 
obtaining a sophisticated electronic control system for touchscreens, voice recognition circuitry, hand-held 
enabling an easy to use control over such home prod- remote controls, computer keyboards or telephones. In 
ucts and systems. Many such products and systems are f^ct, virtually any type of electronic subsystem may be 
also increasingly being used in commercial environ- connected, by means of an appropriate interface, to the 
ments, such as electronic conference rooms, in which it 25 pyg^gj^ system. 

is also desirable to provide an easy to use control system ^he present invention is also compatible with com- 

for use by individuals who are often not skiUed in tech- mercially available automation controllers. Thus, for 

nological arts. example, when a control task requires an extensive 

Known home automation systems are generally bmlt number of inputs and outputs, the system can become 

around a small control box which is connected by 30 "master controller" for a wide variety of commer- 

means of existing household 1 10 volt AC winng to one ^ ^.p^ purpose automation controllers. Such a 

or more modules distributed throughout the home. The capability is not available in any other known home 

applianc^ and/or lights to be controlled are m mm ^^^^^^^^ controller. 

connected to the modules and may thus be controlled innovative expansion capabilities dis- 
from the control console by the homeown^^^ 35 ^ ^^^^ invention, while allowing 
advanuge of such "power-hne earner home conuo^ simultaneousoperationofmultipletypesof user devices 
systems IS that they are low m cost and utilize «ustmg eompatibilhy with commerdal automation control- 
home wirmg^Hdwever, such^^^^^^^^ ^ ^ ^^j, li^,^ 

T^^"^^^^:^^^^^ 40 subsystems and Xith external information retrieval .r- 

In addition, such systems allow the control of only a vices. j - *v u 

relarively Ihnited mnnber of types of electrical appli- . Consumer appliances used m the home are increa^ 

^ces. immely, lights and small electrical appliances. f^y becommg more intelligent. Tlie apphance manu- 

^^do not, however, allow for any sophisticated facturers are mcrea^gly mcorporatmg ^^^^ 

pro^amming functions other than perhaps a time on 45 microprocessor-based mteUigent 

Ld time off feature Thus. power-liTe carrier control sysl«n controller of the present mvenUon «nbodies a 

systems are of relatively limited utility for home auto- multiple data port aipabihty which Provides for its 

mation DUTDOses connection to an unlimited number of such mteUigent 

More s^l^cated home automation system designs appliances simultaneously. For example, the expandable 

are known, which are generally built around a program. 50 home automation system may be connected to control 

mable microprocessor connected to a local bus which or communicate with mteUigent audio/video systems, 

accepts a limited number of input/output control cards. heating/cooling systems, access control systms, secu- 

Such systems may aUow the connection of one or two rity systems, telephone systems, appliances and hghUng 

user control devices such as a keypad or a touchscreen systems. , . ; , , « i. 

for inputting control commands to the home automa- 55 Havmg access to multiple data ports also allows the 

tion system. However, such systems have a predeter- system disclosed in this application to dedicate one or 

mined limited number of how many devices and user more data ports for connecrion to external information 

interfaces the system can support. Generally speaking. services or gateways to information services by means 

in order to expand such systems, a second identical of a modem or other type of data connection. That 

controller b required with its own progranuning for 60 allows the instant inventive home automauon system to 

controlling its own connected devices and user inter- become an information provider as well as a controller, 

faces. Although such an approach may be cost effective SUMMARY AND OBJECTS OF THE 

for small home automation systen:i5, it is too limiting for INVENTION 
more sophisticated automation tasks or for larger 

jjQmes. 65 In view of the foregoing, it should be apparent that 

The mventive expandable home automation system there still exists a need in the art for a home automation 

disclosed in this application overcomes such Ihnitations control system which allows for tiie sophisticated con- 

and drawbacks of the prior art systems by being de- trol, including programming, of virtually any home 
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subsystem or appliance in which elecUonic means are Briefly described, these and other objects of the in- 
utilized by the user in a simple to understand and yet vcntion are accomplished in accordance with its appa- 
precise manner to accomplish the desired control or ratus aspects by providing a home automation control- 
monitoring function. It is, therefore, a primary object of ler which is designed to support multiple numbers and 
this invention to provide an expandable home automa- 5 multiple different types of data communications with 
tion control system for providing sophisticated control external systems. Such a controller is unlimited in its 
functions over complex subsystems found in a home means of communications with external systems since it 
which is characterized by intuitive and simple to use is fundamentally compatible with all forms of data corn- 
user interfaces. mimications. The system controller utilizes a micro- 
More particularly, it is an object of this invention to 10 processor-based computer, such as an IBM AT type, or 
provide an expandable control system which provides compatible, with its associated memory and storage 
for a simple. inUiitivc and easy to use control over com- . sul^ystems. The processor is coimecied by 
plex subsystems utilized in commercial buildings. ^ .data bus to a plui^ity of external parallel 
Still more particularly, it is an object of this invention and semi- mterfaces. external custom interfaces and 
to provide aT expandable home automation comrol 15 external network interfaces. The external paralle and 
system which includes a multiple port, expandable con- f^nal mterfac^ are ^^^^^ 
JLj. ^ terns, such as home automation buses like CEBUS or 

Aether object of the present mvention is to provide SMART HOUSE. If n^ry, optional protocol 

^ vii u™- Jlt^^^^r.^ ^«t^«i c J*<.«, ;« translators are connected between those external sys- 

an expandable home 20 tems and the external serial and parallel interfaces, 

which the user utihzes The external custom interfaces are comiected directly 

touchscreens for mstructmg the system to perform its ^^^^ ^^^^^ ^^^^^ ^^^^ ^^^^^^ ^^^^^^^ 

control ftmctions. interfaces are connected directly to multiple external 

A further object of the present mvention is to provide ^ ^^^^ structure, an 80286-based 

an expandable home automation control system m ^ compatible microcomputer can be utilized, by 

which voice recognition is utilized by the user to m- „jeansof the IBM AT data bus. to control various prod- 

struct the system to control subsystems present m the subsystems within a home or commercial 

home. . ^ , . . building, such as lighting systems, security systems, 

A further object ofthe present mvention is to provide ^^^^ ^^^j^^pj^ external terminals, and to 

an expandable home automation system m which voice 3^ ^^^^^ of commands by a variety of means such 

recognition may be used m concen with high resoluuon ^ touchscreens, voice recognition, telephones, custom 

color graphics displays m order to provide the user with switches or any device capable of providing input to a 

an easy to use interface for instructing the system to computer system. 

perform its control or monitoring functions. jjjg method ofthe present invention is carried out by 

Still another object of the present invention is to 35 ^i,e use of a Home Environment Automated Real-time 

provide an expandable home automation system which Transactor (HEART) which is a standardized, modu- 

incorporates multiple types of user devices which may software program that is configured for each instal- 

be utilized simultaneously. lation. Secondary processors are utilized, under 

A still further object of the present invention is to HEART control, to relay information to the central 

provide an expandable home automation control system 40 80286-based microcomputer or central processor, or to 

which uses dynamic, object-oriented touch or cursor- translate central processor commands into commands 

controlled displays for controlling and scheduling ac- ^^j^ dedicated devices can imderstand. A star hard- 

tions. ware topology is currently utilized. The secondary 

Still more particularly, it is an object of this invention processors manage the voice recognition and voice 
to provide an expandable home automation control 45 synthesis subsystems, telephone communication subsys- 
system which integrates tailored information retrieval tems, touchscreen subsystems, hand-held remote con- 
together with communication services. trol imit communications, input/output control and 

Still another object of the present invention is to monitoring, security/fire safety and monitoring and 

provide an expandable home automation control system other intelligent subsystems such as lighting, audio, 

which displays plan views of the home and allows the 50 video and HVAC. 

user to create certain moods therefore. The central processor's HEART software is com- 
A still further object of the present invention is to prised of six major portions: the Subsystem Interrupt 
provide an expandable home automation control system Handler, the Internal Scheduled Interrupt Server 
which provides multiple dimensions of interaction and (ISIS), the Polling Loop, the Tasks, the Device Drivers 
control. 35 and the Support Environment. Those modules interact 
Another object of the present invention is to provide with any software contained in the intelligent second- 
an expandable home automation control system which ary processors in order to provide an integrated and 
utilizes multifunction display monitors to display multi- seamless control environment for the user, 
pie types of video imagery. With these aiid other objects, and advantages of the 
A further object of the present invention is to provide 60 invention that may become hereinafter apparent, the 
an expandable home automation control system which nature of the invention may be more clearly understood 
utilizes an electronic interface to a multi-zone security by reference to the following detailed description of the 
system to thereby allow touch or cursor-control of the invention, the appended claims and to the several draw- 
security system by means of graphics displays. ings attached herein. 

praerSSuSbt hl?L=i„t^^^^^ " BRIEF DESCRIPnON OF THE DRAWINGS 

in which spoken messages are utilized as prompts for FIG. 1 is a schematic block diagram showing the 

touchscreen use. apparatus of the present invention; 
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no. 2 is an schematic block diagram of the security FIG. 6is a How chart showmg part of the Task 4 

sysxcm interface used as part of the apparatus of FIG. 1; module, the operation of the. lighting moods menu, of 

FIG. 3a shows the main menu touchscreen display the present invention; , 

used unth the present invention; FIG. 6rf is a flow chart showmg part of the Task 4 

FIG. shows an audio/video sub-menu touchscreen 5 module, the operation of the dormant screen, of the 

display which may be utilized with the present inven- P^^"^ invention; ^ r -r i ^ 

. ^ ^ FIG. 6e is a flow chart showing part of the Task 4 

"fig. 3c shows a VCR sub-menu touchscreen display module the interaction of the main menu and dormant 

j-jy. ^ aiiw ^ . , , . screen funcuons. of the present mvenuon; and 

which may be utdized w«h the pr«ent mvent.on; ° ^ ^ ^ 4 

FIG. 3rfshows a CD se ect,on sub-menu touchscreen ^| 

display which may be utihzed, with the present mven- p.^^ i„^ention. 

t*o^ J. V FIG. 7j is a flow chart showing the validation routine 

FIG. 3^ shows a distributed audio sixb-mcnu touch- ^^^^ touchscreen task of the present invention; 

screen display which may be used with the present ^ ^ showing the immediate re- 

invention; sponse portion of the master touchscreen task module of 

FIG. 3/shows a distributed audio location sub-menu ^^^^ invention; 

touchscreen display which may be used with the pres- fcisz flow chart showing the neutral state of 

cnt invention the master touchscreen task module of the present in- 

FIG. 3^ shows a security management sub-menu vention; 

touchscreen display which may be used with the pres- pjQ 7^ is a flow chart showing the general contex- 

ent invention; tual state of the master touchscreen task module of the 

FIG. 3A shows a first state of the entry level floor present invention; 

plan sub-menu touchscreen display which is accessed as jt^q 7^ jg g flow chart of the display cursor subrou- 

a function of the security management sub-menu of 25 tine of the master touchscreen task module of the pres- 

FIG. 3^; ent invention; 

FIG. 3/ shows a second state of entr>' level floor plan FIG. If is a flow chart of the print zone status subrou- 

sub-menu touchscreen display of the entry level which tine of the security floor plans sub-menu of the master 

is accessed as a function of the security management touchscreen task module of the present invention; 

display screen of FIG. 3^* 30 FIGS. Ig-ln form a flow chart of the security floor 

FIG. 3y is a sub-menu touchscreen display of a pass plan sub-menu of the master touchscreen task module of 

code keypad which is utilized by the user to enter a the present invention; 

password into the system of the present invention; FIG. To is a flow chart of the redormant or return to 

FIG. 3Jt shows a third state of an entry level floor dormant routine of the master touchscreen task module 

plan sub-menu touchscreen display of the entry level 35 of the present invention; and 

which is accessed as a function of the security manage- HG. TP is a flow chart showing the reset synchrom- 

ment display screen of FIG. 3g: ^^on routine of the master touchscreen task module of 

FIG. 3/ shows the security event log display screen of the present invention, 

the security event log function of the security manage- FIG. 8^ is a flow chart showmg the validation routine 

ment disnlav screen of FIG 3g; ^ secunty/fire task of the present mvention; 

HG 3tlh^^^^^ sub-menu touch- . FIG. 86 is a flow chart showmg the j^^^^^^^^ 

screen display which i^y be used with the present immediate response, neutra^ and contextual states 

screen ui!»piay wiii^u u^y K portions of the secunty/fire task of the present mven- 

mvention; and ^ , 

no 3n shows the "Jonna^on ^l^r 45 FIG. 8c is a How chart showing additional contextual 

screen display which may be used with the present functions of the security/fire task of the present 

invention. invention 

FIG. 4<; is a flow chart ff^^S ^he Eternal Sched- ^ ^ ^^^^^ graphics-oriented 

uled Intemjpt Server module of the HEART central ^^^^^ cursor-controlled event scheduling subroutine 

processor software utilized with the present mvention; ^ ^^^^ invention. 

^ . ^ c - . , FIGS. lOo and lOg are graphics touchscreen repre- 

FIG. 46 is a flow chart showmg the Sena! Interrupt gentations of a portion of a touchscreen utilized by the 
Handler module of the HEART central processor soft- instant system to schedule a day and month for opera- 
ware utilized in the present invention. the user selectable functions; 

FIG. 5fl is a flow chart showing part of the Polling 55 pjQ ^ 3 graphics touchscreen representation of 

Loop module of the HEART central processor soft- ^ portion of a touchscreen utilized by the instant system 

ware utilized with the present invention; showing a summary of the dates and times that have 

FIG..56 is a flow chart showing the second half of the been selected usmg the graphics scheduling tools of 

Polling Loop module shown in FIG. Sa; and FIGS. tOa, 10(g, 10c and lOd; 

FIG. 5c is a flow chart showing the voice recognition ^ piG. 10c is a graphics touchscreen representation of 

function of the Polling Loop module of the present a start dock which may be utilized to set the start time 

invention. for a function selected by the user of the system; 

FIG. 6fl is a flow chart showing part of the Task 4 FIG. lOd is a graphics touchscreen representation of 

module, the touchscreen functions, of the present inven- a finish clock which may be utilized to set the stop time 

tion; 65 for a function selected by the user of the system; and 

FIG. 66 is a flow chart showing part of the Task 4 FIGS. lOe and 10/ are alternative graphics touch- 
module, the operation of the main menu of the present screen represenutions of the start and fmish clocks 
invention; shown in FIGS. lOc-lOii 
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FIG. 11 is a flow chart showing the checktouch func- 
tion of the master touchscreen task module of the pres- 
ent invention whose function is to compare the touch- 
points of the user with the touch boxes or menu. 

FIGS. 12a-X2g show an example of a series of touch- 5 
screen menus which may be utilized to control an elec- 
tronic conference room or an entertainment media 
room. 

FIG. IZa shows a flow chart of the screen time-out 
subroutine of the ISIS events module for use with the 10 
present invention; 

FIG. 136 shows a flow chart of the security response 
expected subroutine of the ISIS event module of the 
present invention; 

FIG. 13c shows a flow chart for the wait for bad 15 
password subroutine of the ISIS event module of the 
present invention; and 

FIG. 13d shows a flow chart of the complete security 
message subroutine of the ISIS event module of the 
present invention. 20 

FIG. 14 shows a flow chart of the secondary touch- 
screen task module for use with the present invention. 

FIG. ISfl shows a flow chart of the validation routine 
of the input/output task module of the present inven- 
tion; 25 

FIG. 15b shows a flow chart of the immediate re- 
sponse and neutral stale routine of the input/output 
controller task module of the present invention; 

FIG. 15c shows a flow chart for the contextual state 
routine of the input/output controller task module of 30 
the present invention; and 

FIG. ISd shows a flow chart of on-to-ofi" and ofF- 
to-on routines of the input/output controller task mod- 
ule of the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 

Referring now in detail now to the drawings wherein 
like parts are designated by like reference numerals 
throughout, there is illustrated in FIG, 1 a block dia- 40 
gram representation of the expandable home automa- 
tion system of the present invention. As previously 
described, the instant expandable home automation 
system is built around an 80286-based microcomputer, 
such as an IBM AT class microcomputer, or compati- 45 
ble. Obviously, an 80386-based microcomputer or any 
computer equaling or exceeding the computing speed 
or processing power of an IBM AT class microcom- 
puter may be used with the present invention. The mi- 
crocomputer together with its random access memory 50 
(RAM), expanded RAM memory and associated hard 
disk drive and power supply is shown in FIG. 1 as the 
central processor 10. In the preferred embodiment, the 
central processor includes 3-4 megabytes of extended 
RAM and a hard disk having a storage capacity of 55 
approximately 20 megabytes. 

The central processor 10 is connected, by means of an 
IBM AT dau bus, or its equivalent, through a plurality 
of standard or custom interfaces to either control each 
of the subsystems automated within the home environ- 60 
ment or to transmit or receive either data or instructions 
from within the home environment. For example, for 
control purposes, the central processor 10 is connected 
by means of the data bus 12 to a multiple serial interface 
14 which in turn is connected to two touchscreens 16a 65 
and 16b. As will be described later herein, the user can 
instruct the home automation system, by means of the 
touchscreens 16a and 166 to carry out a particular task. 



Under those circumstances, the central processor re- 
ceives the input from the appropriate touchscreen after 
it has been processed by the serial interface 14 and then 
processes the appropriate instructions to perform the 
selected task. As many touchscreens 16^ and 166 as are 
desired can be distributed throughout the automated 
environment. In order to increase the number of touch- 
screens that may be used to control the automation 
system or to add additional devices requiring a serial 
interface, additional multiple serial interface cards may 
be added. The multiple serial interface 14 may prefera- 
bly be a DigiBoard COM/8 serial interface, available 
from DigiBoard, Inc. of Saint Louis Park, Minn. The 
touchscreens 16fl and 16b may preferably be Elograph- 
ics Accutouch touchscreens, available from Elograph- 
ics of Oak Ridge, Term. 

In addition to controlling the instant automation sys- 
tem by means of one or more touchscreens 16a and 16b, 
the central processor may also be connected to a stan- 
dard IBM AT or compatible computer keyboard 18. 
While such a keyboard may be utilized by the user, it 
will more likely be utilized by technical . personnel dur- 
ing set up and maintenance procedures. A standard PC 
computer keyboard may also be efiectivcly utilized. 

A receiver for a hand-held remote control unit may 
also be connected to the serial interface 14 so that the 
user may be provided with a hand-held infrared, RF or 
other type of remote 22 for commanding the central 
processor to perform the various available tasks. 

The central processor 10 may also be connected by 
means of the bus 12 to a first parallel interface 24<7 
which may be utilized to both receive and to output 
direct digital signals. A wide variety of switch control 
devices may be connected through this parallel inter- 
face 24a, either directly, or through a process controller 
26, to the switch control devices. A preferred example 
of the switch control devices is the LiteTouch 1000 
lighting switch, available from LiteTouch of Salt Lake 
City, Utah. The switches 2$a and 2Bb can each provide 
for the control of a plurality of devices, preferably nine, 
and are connected directly to the process controller 
26a, which, for this example, may preferably be a proto- 
col translator such as an opto-isolated buffer. Such a 
buffer is available from Industrial Computer Designs, 
Inc., of Westlake Village, Calif., as the AOISO-8 eight- 
channel output driver board. The parallel interface 24a 
may preferably be an ICD1018 multiple parallel port 
board, which is also from Industrial Computer Designs, 
Inc. 

The central processor 10 may also be preferably con- 
nected to a second parallel interface 24^ by means of the 
data bus 12. The parallel interface 24b communicates 
directly with a process control system 26b made up, for 
example, of a Metrabyte relay input/output board, such 
as the Model MEM-32 electromechanical relay board, 
an analog input board such as the MAI 16 A/D board, 
or a digital input/output board such as the MSSR-32 
solid state relay module board, all available from Met- 
rabyte Coip., of Taunton. Mass. The parallel interface 
24^ may preferably be a Metrabyte MDB-64 controller 
board. 

Both the relay output board and the input/output 
board may be connected to the electrical appliances or 
devices 31 such as door locks, security gates, lawn 
lights, speakers, or any other switch-controlled device. 
The relay output and input/output boards may also be 
connected to the plumbing related systems 33 such as 
baths, showers, faucets, pools, spas, and fountains. The 
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analog board may be connected directly to the analog the user with information regarding the operation of the 

sensors 29. which provide a voltage output indicating, home automation system. 

for example, temperature, humidity, pressure, light Voice recognition circuitry may also be connected by 

level, distance, vibration, air quality, or any other useful means of the bus 12 to the central processor 10. A 

parameter for automation purposes. The input/output 5 speech processor 58, such as, the Model TISPEECH 

board may also be connected to the digital sensors 27, available from Texas Instruments, Dallas, Tex., may be 

such as security sensors, pressure mats, driveway sen- connected to one or more remote microphones 64, such 

sors, status relays, or other digital indicator devices. as the Crown PZM microphone, available from Crown 

The security system 38 used in the automated envi- International, Inc., of Elkhart, Ind. Such remote micro- 

romneni, may preferably be a Silent Knight Model 4000 10 phones provide a means by which the user of the pres- 

security system, available from Silent Knight, of Minne- ent expandable home automation system caa communi- 

apolis. Minn. Such a home security system 38 is con- cate with the system by requesting tasks by voice. The 

-nected to an RS-232-to-security system translator 36 home automation system may provide voice communi- 

which is in turn connected to the central processor 10 cations to the user by means of one or more remote 

by means of the serial interface 14 and a data bus 12. It 15 speaker 66, which in turn are connected to the speech 

can provide simultaneous RS-232 communications with processor 58. 

eight separate external systems. When thus configured, addition, the central processor 10 of the instant 

the serial interface 14 can communicate directly with home automation system may be connected to a home 

any external system, such as an Biographies touch- telephone system 62, such as the Panasonic KX-T61610 

screen controller, or any other system that uses stan- 20 j^^y system, or a single line standard telephone system, 

dard RS-232 protocols. If, as is the case in the preferred ^^y „jeans of a telephone interface 60. Such telephone 

embodiment, the external device uses a nonstandard interface may preferably be a Model TISPEECH 

protocol, then an RS-232 protocol converter, such as phone Interface available from Texas Instruments; 

element 36. can be used to convert between the RS-232 yj^^ j^^^^ automation system, schematically shown 

protocol and the protocol utilized by the device to be 25 ^ achieves an unlimited expandability and com- 

controUed. That same interface, as is shown as element patibUity by using multiple readily available (in roost 

14 in FIG. 1, may also be used to connect the centra instances) subsystems that can be added without limit, 

processor 10 and its data bus 12 through a protocol function is achieved by the use of multiple fiexible 

converter 34. ^L^f^^;^^^'''^^^^ ^ standard mterfaces for controlling and commmiicating 

buses such as CEBUS and SMART HOUbt. 30 ^ external devices and subsystems as 

As wUl probably be clear to those of ordinary skiU m ^ translating modules which are used to cominuni- 

the art, the design of the invenUye expandable home ^^^^ ^ ^^^^^ nonstandard 

automation system describe! ^'''^ ^f^.J^^^^l"^ communication links. A more detailed description of 

ofmterface boards that are designed to te^^ the hardware implementation of the present expandable 

patiWewxththebusU ofthecenU^^ home automation system follows, 

interface boards may then be literally pjugged in in . ' 

order to provide control and communications with an TOUCHSCREENS 

external system. An example of the mx of Mich directly invention utilizes a high resolution color 

compatibleinterfacelK^ards g^^S^ch^reen to control cert^ specific features 

be a Multitech Modd 212 modem which « conne«ed « ^ P , expandable home automation system, 

between the bus 12 of the central processor W and any o. ^e J^^^^ automation systems that 
«tenud sj^tems with 1? "tflize touchscreen control are presently known, such 

be esabliAed using a telephone Ime, such as a remote ^^^^ character-based graphics systems. 

L^er example of such a directly compatible inter- 45 low resolution color ^ics sy^t«>s. or monocluram^ 
fa^S fcX analog input interface 46" which pro- graphKS systems. Such systems attempt to "^"f 
^"es^interconnection bitween the central processor cost of Ujesystem whde at the same ^« 
10. by means of its bus 12. with a plurality of analog dfP^y speed. Such ff^phics systena cost le« per d^ 
s«sors4&, and 48& Such analog input interface 46 may play screen by mimmmng tibe use of cotor or by uang 
^mbly tea Microtech PC Di(tecl.avaaable from 50 graphics characters mstead of full high resolution 
Microtech. ofLonghome, Pa. Examples of such analog graphics screens. ^ «• • . r 
Ss ar; those which monitor outside temperature. Whi^e such method of d^play may be efficient for 
tath water temperature etc some limited home automation systems, it is only opera- 
It is also posdble with the present inventive expand- ble under circumstances where there are only minimal 
able home automation system design to comiect to a 55 commands that «in be given through such a graphics 
completely external network such as Ethernet, by dkplay. Where the automation system is more capable, 
means of an Ethernet network interface board 50. Using it is useful for the touchscre« duq>lays to copveV more 
such an interconnection, a wide variety of applications information as weU as to effecnvdy highlight touch 
such as information retrieval and remote home automa- sensitive areas on the touchscreen. ^ ^ ^ ^ ^ 
tion control can be achieved, for example, utilizing 60 The present invention mcorpoiates both the hard- 
multiple external terminals 52a 52*, 52f or the network ware and software necessary to provide scr^resolu- 
fde server 51, conneaed to the Ethernet network inter- tion ranging froni 640x 350 pixels up to 1024X768 
face50 whichisitsdfconnectedby meansof thebusl2 pixels with from 16 to 256 colors. Thus, dependmg upon 
to the central processor 10. the desired application and type of screen desired to be 
The central processor 10 may also be connected, by 65 displayed, the touchscreen display can be provided with 
means of its bus 12, to a graphics interface 54. such as an as much resolution and color as necessary. Such display 
EGA or VGA interface, which is in turn connected to flexibility allows the present home automaUon system 
a respective video display monitor 56 which provides to incorporate powerful graphics features for ease-of- 
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use, such as icons, pop-up uindows. color-coding, di- Other functions which can be performed by the home 

rect manipulation and detailed floor plans. automation system involve the filling of baths, pools 

It also allows the use of a wide range of colors in and spas and the maintenance of a desired temperature 

order to enhance the aesthetic appearance of the touch- in those facilities, as well as the control of any pumps 

screen displays and in order to color coordinate them 5 associated with those facilities. The home automation 

with the surroundings in a home, commercial or other system can also control individual devices and appli- 

cnvironment. Furthermore, as additional features not ances such as kitchen appliances, exhaust fans, humidifi- 

presently available become available on newer versions ers. and dehumidifiers. It can control motorized devices 

of graphics cards, such as animation and video mixing, such as skylights, draperies, furniture, walls, screens, 

the standard bus structure of the present invention wiU >0 ceiUngs. awnings, physical security barriers, door locks, 

allow those features to be readUy and easDy added to others. The home automation system can also con- 

the home automation system. trol answering machines, voice mail systems, and pro- 

The preferred embodiment of the present invention vide inamtenancc reinmders. Of course, all of these 

incorporates as the graphics interface 54 the ATI VIP control funcuons can be ext«ided to similar short term 

graphi^ hardware, available from ATI Technologies 15 residential apphcations, such as boats aircraft, office 

of Ontario, Canada. Under control of the system soft- smtes^ conference rooins, auditonums. classrooms^ 

ware, to be later described, this hardware can dynami- hotels hospitals, and retirement homes. They a^so 

cally display graphic screens under either the EGA offer mmiediate application to the mfirm or disabled 

standard (640X350 pixels X 16 of 64 possible colors) or _ P°P^f"°?)- , . , , . „^ u .u 

the VGA standard (640X480 pixelsX256 of 262,144 ^ °^f^ ^^^^ ^ controlled by the 

*bl I rs^ disclosed touchscreen are sophisticated secunty sys- 

po^D e CO o ^ ^2 zones, ice and snow removal by 

The present home automation system IS also hard- energizing, for example, heaters built into sidewalks and 

ware compatftle witii many other readily avail^k drive^yf when pr^pitation is detected and the out- 

graphics boards which offer extended capabihties^ For tem^^ture drops below a certain predetermined 

example, by mcoiporating a different graphics card and ^Hocked doors, the initiating 

makmg nwnor modifications to the home automation tS^phone calls^nd the logging of incoming and 

system software, the present system can provide graph- telephone calls. Information retrieval can also 

ICS up to 1024X768 pixels. Another readily av^ble ^ ^ccomplidied utilizing the instant touchscreen as 

card offers compaubility vnth the standard NTCC 3^ ^^jj ^ jj^^ throughout a home by which, 

video format used m Unit^ Slates television sets. TTie ^^^^^ touchscreen command, certain predetermined 

system may also mclude high performance graphics settings for lights throughout the home can be readily 
display processor cards for advanced graphics effects 

including animation. In f^ct. any system in a home or other environment 
Using the high resolution graphics discussed above m 35 that can be automated can be effectively automated 
concert with high resolution touchscreens. many func- utilizing the high resolution graphics displays described 
tions in a home or other environment can be controlled. ij^^ein in concert with high resolution touchscreens. 
For example, audio/video entertainment systems can be Such systems, in addition to those described above, 
actuated and caused to perform all of the functions ^^^^ include telephone answering, controlling foun- 
available by use of their respective remote controls. ^ tains or in-ground sprinkler systems, controlling kitchen 
This feature is accomplished by means of an infrared othej. appliances, controlling motorized drapes, 
emitter device which is utilized, imder control of the windows and skylights and the scheduling of many of 
centralprocessor 10. to input to the audio/video device these functions and those functions previously de- 
being controlled the appropriate code for controlling scribed. In addition, local information display, such as 
the desired function. Alternatively, a direct serial link 45 time, temperature, etc., can readily be accomplished 
or other electronic interface could also be used to con- utilizing the instant touchscreens. 
trol the audio/video device. For example, the present 

home automation system can, utilizing a touchscreen, VOICE RECOGNITION CIRCUITRY 

energize a connected standard television receiver and The use of voice recognition is an ideal user interface 

control the volume, select the channel, select the stereo 50 device for a home automation system because it allows 

or mono mode, and, as described, control any function the user to control functions in the home simply by 

which can be controlled using that television receiver's speaking. Voice recognition is also useful in certain 

remote control. commercial environments. The use of the high resolu- 

Audio entertainment equipment, such as remotely tion graphics display together with voice recognition 

controllable receivers, amplifiers, turntables, cassette 53 allows the present home automation system to provide 

players and compact disk players, as well as video cas- high quality visual cues to the user of the present con- 

sette recorders, can be controlled in the same manner. trol options, feedback of whether the user's command 

For example, a certain selection or series of selections phrase has been recognized and the results produced for 

can be made, using the touchscreen, for a compact disk. the user as the result of a commanded function. 

An AM or FM station can also be selected from the 60 One of the difficulties with the use of voice recogni- 

touchscreen menu. In addition, the user can use any tion in home and other environments is that such sys- 

touchscreen to distribute audio or video signals to any terns must perform under at least low level background 

one or more high fidelity speakers or monitors con- noise and must be able to extract key words or phrases 

tained in the home or other environment. out of a speech or general noise backgroimd. The inven- 

The instant home automation system also utilizes 65 tive expandable home automation system meets these 
video monitoring cameras so that, for exaniple, the user performance specifications by the use of certain hard- 
can elect to view the signal from such cameras on se- ware, together with software to enhance the perfor- 
lected video monitors throughout the house. mance of the voice recognition functions in the home 
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environment, as well a high resolution graphics display high resolution color graphics display 56, immediate or 
sj^j^ scheduled control of many home automation features 

The implementation of the voice recognition system can be accomplished. Such features include direct con- 
described herein may be accomplished by means of the trol of the lighting within the home, or, if preferred, 
central processor 10 and its interconnection along bus 5 controlling the lighting mood (i.e. choosing from one of 
12 to a speech processor 58 which in turn is connected a predetermined number of preset lighting levels) either 
to a remote microphone 64 and a remote speaker 66. in one room or throughout many rooms of a home. 
The central processor 10 may be. for example, an AT Such a system can select complete living moods, which 
compatible 80286 based microcomputer. Model CAT- consists of macros of multiple conmiands. for various 
902 from Diversified Technology, of Jackson, Miss, and 10 lighting, temperature, open or closed draperies, skylight 
a Model CHI 626B AT-compatible passive backplane settings, and entertainment system settings that have 
system bus, also available from Diversified Technology. . been predetermined and stored as data files. 
As previously discussed, a hard disk storage device is The voice recognition system, in connection with the 
connected to the microcomputer for permanent storage high resolution color graphic display 56, can also pro- 
of trained recognition vocabularies. In addition, an ex- 15 vide home appliance control as well as security system 
tended RAM memory, as previously described, is also control for components connected to the central pro- 
provided for connection to the microcomputer to pro- cesser 10. as has been previously described. In addition, 
vide rapid loading of new vocabulary data. the audio/video entertainment system controls and 

The speech processor 58 may preferably be embodied audio distribution network described above ni connec- 
in the present system by one or more Texas Instruments 20 tion with the use of only a voice recognition system can 
TISPEECH speech processing boards for IBM and obviously be controlled with the both the high resolu- 
compatible personal computers. The system software is tion color graphics display and voice recognition sys- 
available from Texas Instruments as Model TIS- terns. 

PEECH, which performs the basic voice recognition. Additional functions which can be controlled with 
vocabulary loading, and microphone control functions. 25 the combination of voice recognition and high resolu- 

Each TISPEECH board is connected to a single tion color graphics displays are the security camera 
crown PZM microphone for voice pickup, shown as switching and control, security camera function and the 
remote microphone 64. complex security system 38, as previously described 

In order to effectively utflize the voice recognition above. Also, local and remote information retrieval, 
system of the present invention, a graphics interface 54, 30 such as trnie and weather and access to various remote 
which may be a Model VIP VGA video graphics dis- data bases can be achieved using the voice recognition 
play generator, available from ATI Technologies, is and high resolution color graphics display combination, 
connected directly to the central processor 10 by means Control of the locks for various doors of the home as 
of the bus 12. The graphics display interface 54 is used well as bath and spa controls, for example, to run the 
to drive a video display monitor 56, which may prefera- 35 bath, set the temperature and turn on any pumps or 
bly a Mitsubishi AUM-1381A high resolution color motors, as previously described, can also be achieved 
graphics display monitor. A separate video graphics by the use of the voice recognition system and high 
display generator and high resolution color graphics resolution color graphics display system combination, 
display monitor are provided at each voice recognition In addition, the telephone system can also be controlled 
area in the home, together with a separate microphone 40 by that combination. 

and speaker to effectuate control and aur^ feedback. MULTIPLE TYPES OF USER INPUT DEVICES 
Also, as described later herem, a portion of the software 

which operates the central processor 10 is dedicated to The present home automation system, as has been 
optimizing the voice recognition parameters of the previously described, by means of its extended bus and 
speech processor 58 in order to maximize performance 45 ability to interface with a wide variety of devices, can 
in a home automation or other environment. support any user device that can be controlled serially. 

Using such a system, the user's voice can be utilized controlled with a parallel port, or with a custom inter- 
in the present home automation system to provide ei- face compatible with th^ standard AT bus used by the 
ther immediate or scheduled control of complete living inventive home automation system. Since both the bus 
moods, which are comprised of macros of multiple 50 and number of serial or parallel ports can be increased 
commands, audio/video entertainment system control, as desired, the number of interface devices and thus the 
such as selection of the desired audio or video source, number of devices in the home environment or corn- 
selection of the source parameters, such as channel mercial environment that can be controlled can also be 
selection, play, fast forward, rewind, volume, base, increased as desired. That allows the disclosed home 
treble, balance control, etc., as well as the audio distri* 55 automation system to be configured to meet the exact 
bution network, for example, selecting which set of needs of each home owner or business user. The system 
speakers the audio or video system will send its output can thus be tailored to the layout of the environment to 
to, as well as providing a muting function for each of be controlled. For example, a touchscreen display can 
those sets of speakers. In addition, the system can also be provided in the kitchen, a touch switch with nine 
both control and switch between a plurality of security 60 buttons can be placed near the exterior doors, voice 
video cameras, including panning, tOting, zooming of recognition can be installed in the master bedroom and 
such security cameras. Fmally, the voice recognition any inade telephone can be used to control the system, 
functions of the present home automation system can be It should also be remembered that an important as- 
utilized to control a complex home security system 38 pect of the present invention is the commonality of use 
having over 32 zones, which is connected to the central 65 between input devices. For example, both the computer 
processor 10. keyboard 18 and the voice recognition system use the 

When the voice recognition functions of the present touchscreen displays for cues and visual feedback, 
home automation system are utilized in concert with the Wall-mounted telephone touch-tone buttons can be 
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used to control a subset of the touchscreen commands. time Iine» or otherwise select, in a dynamic, graphics 
In addition, although not shown specifically in FIG. 1, manner, a date or time to schedule a home automation 
other user interfaces may be added to the home automa- function. This type of system can also be utilized with 
tion system, such as track balls, mice, joy sticks, light other stand-alone systems such as lighting systems, en- 
pens and other cursor control devices. In addition, 5 lertainment systems, security systems, energy manage- 
while a hand-held remote control tmit is presently used, ment systems and others. 

wireless versions of any of the present interfaces shown Functions that can be scheduled in this manner in- 

in FIG. 1, such as wireless portable touchscreens and elude turning a light or appliance on or off, watering the 

wireless microphones or radios for remote voice recog- lawn, turning on different lighting moods at different 

nition, may also be used. Voice recognition may also be 10 times of the day, turning on audio music or a television 

used over the tel^hone when that technology becomes station, VCR recording of television programs, operat- 

available. ing modes or features of the home automation system 

In order to implement the use of multiple types of itself, operating modei of a security system, electric 

user devices in a modular manner which allow for dif- locks, reminders for the system user, operating modes of 

ferent types of devices in different parts of the home to 15 a telephone system and automatic information retrieval, 

be used to control the home automation system de- As will be described further herein, the same types of 

scribed herein, the following equipment may be simulta- displays may be used directly to control entertainment 

neously connected to the home automation system cen- systems by working with a graphics representation of 

tral processor: (1) a standard PC computer keyboard 18; the entertaimnent system and its distributed speakers 

(2) plurality of dry contact switches 2Sa and 28^ may be 20 and sources. The user controls the system by selecting 

connected to the central processor 10 by means of a his audio or video source from a graphics display and 

solid state input module anid a pull up resistor to a Met- then selecting the rooms in which he would like the 

rabyte MSSR-32 solid state relay board which is shown audio and video played. 

as process controller 26b, which is in turn connected to This particular feature of the inventive home automa- 

the parallel interface 246. which may be an MDB-64 25 don system is implemented primarily as a sofbvare func- 

parallel controller connected to the AT bus 12 of the tion of the instructions run by the central processor 10. 

central processor 10; (3) multiple Biographies Ac- It is communicated to and receives instructions from the 

cutouch touchscreens 16a and 16^ connected serially to user by implementation on the high resolution color 

the central processor 10 by means of the Digicom graphics display monitor 56. First, a background screen 

COM/8 multiport serial interface 14; (4) a plurality of 30 is displayed on the monitor 56 with the appropriate 

hand-held remote receivers 20 may be serially con- graphical representation of the scheduling system, for 

nected to the central processor 10 by means of the serial example, a monthly calendar page as shown in FIGS, 

interface 14; (S) a telephone system 62 with a plurality 10a or 10^ a start clock as shown in FIG. 10c and a 

of hand sets, or a single line standard telephone system finish clock as shown in I^G. lOd Using a touchscreen 

may be coxmected to the central processor 10 through a 35 16 or other cursor control device such as a mouse or 

telephone interface 60 and the speech processor 58 and light pen. the user can dynamically select and/or move 

the AT bus 12; (6) multiple voice recognition locations graphics elements to indicate a date and time to start 

may be connected to the central processor 10 through and stop any action the home automation system is 

Crown PZM microphones or other remote microphone capable of performing. 

systems 64 which are in turn connected to respective 40 One way in which this feature may be implemented, 
TISPEECH speech processors 58 running standard using the display shown in FIG. 10a, is as follows. The 
voice recognition software which themselves are con- user first touches a date on the calendar shown in FIG. 
nected by the AT bus 12 to the central processor 10; and 10a A colored box is drawn around the date to high- 
(7) multiple voice response locations with remote speak- light it. Next, the user touches the hands of the start and 
crs 66 which provide spoken information and instruc- 45 finish clocks to drag them to the desired start and stop 
tional cues to the user. All of the foregoing equipment is times. Finally, the user touches the '*OK" box to con- 
simultaneously connected to the central processor 10 firm and activate the scheduled event, 
and may be used to perform a plurality of functions. Examples of the graphics screens through which the 
-w.^, w ^ r,^« ^^^-n^ T *-T^ user of the instant home automation system may control 

50 the functions of the system are sho>^ in FIGS. 3a-3n. 

GRAPHICS DISPLAY ^ ^^^^ ^ preferred embodiment of the main 

Oneof the most important functions of a home auto- menu screen which provides a plurality of different 
mation system is the scheduling of actions in the home functions which may be called from the main menu, 
environment to be performed automatically by the sys- Such functions include control of the audio and video 
tem at a future date and time. The home automation 53 functions of the home automation system, security man- 
system described herein incorporates dynamic, object- agement functions, lighting moods, information, such as 
oriented^ touch or cursor-controlled graphics displays retrieval from remote data bases, environmental con- 
for effectuating these scheduling functions. trol, such as temperatures throughout the home etc.. 

One way in which the scheduling functions are ac- convenience features such as calling for a bath to be 

complished is through the use of a dynamic graphics 60 drawn and a certain temperature to be maintained and 

representation of clock faces, either 12 or 2 hour, calen- the system configuration functions, 

dar pages, either weekly, monthly or aimually, time FIG. 3b shows the first sub-menu which appears on 

Hnes, or other pictorial representations of day, date the touchscreen when the audio/video fimction box is 

and/or time as a data input/output display for entering selected. When the user places his finger inside the 

and reviewing schedules in a home automation system. 65 appropriate box, the system determines which function 

A touchscreen 16a or 16^ or other cursor control de- has been selected, highlights that function box by, for 

vice, is used to move the hands on the clock, select a example, changing the color of the box. and then dis- 

day on the calendar, move an **on-flag** to a point on a plays the first (and sometimes only) sub-menu associ- 
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ated with that function. As shown in each of the FIGS. 
Za through 3n, each of the touchscreens also contains 
two feature boxes in common» namely the Back and 
Quit function boxes. The back function box, as the name 
implies, functions to take the user back to the previous 5 
touchscreen display. The quit function box, on die other 
hand, functions to end a session of using the touchscreen 
and return the system to a dormant mode which may 
also display a custom introductory graphics screen or 
simply a blank screen which, when touched, will again 10 
cause the system to display the main menu screen. 

The audio/video touchscreen sub-menu shown in 
FIG. Zb presents the user with the various functions 
which can be controlled by the present home automa- 
tion system as disclosed herein. The system can control 15 
a VCR, a television, select certain AM or FM presets on 
a receiver, control a compact disk, distribute audio 
throughout the home, be used to operate the volimie, 
lone and other controls associated with the audio and 
video equipment controlled from the audio/video 20 
touchscreen, and turn the audio/video system off. Obvi- 
ously, additional similar pieces of equipment and func- 
tions may likewise be controlled. 

FIG. 3c shows the VCR sub-menu touchscreen 
which is displayed if the VCR function is selected from 25 
the audio/video touchscreen shown in FIG. Sb, All of 
the functions which can be remotely controlled on a 
VCR, in addition to scheduling the VCR, can be con- 
trolled from the VCR touchscreen of FIG. 3c. although 
only seven frequently used functions are displayed in 30 
FIG. 3c Thus, using FIG. 3& the user can cause the 
VCR to play, to stop, to operate in the fast forward 
mode, the rewind mode, the slow-motion mode, the 
freeze/frame advance mode or the user can schedule 
the VCR to operate and record at a predetermined lime. 35 
By selecting the VCR scheduling function, the schedul- 
ing screens shown in FIGS. lOo through lOd would 
appear on the touchscreen for the user to schedule the 
VCR event. 

FIG. 3^ shows the CD selection sub-menu touch- 40 
screen option which may be selected from the sub-menu 
touchscreen of FIG. 3& £ach of the selections which 
are currently available for play by the CD player are 
shown in the menu for selection by the user, as well as 
the CD controls function. The CD controls function 45 
allows the user to control the compact disk player using 
the touchscreens of the home automation system. 

FIG. 3e shows yet another sub-menu touchscreen 
which can be accessed from the audio/video sub-menu 
touchscreen of FIG. 3& The touchscreen shown in 50 
FIG. 3e allows the user to select an audio source which 
may then be distributed, selectively, throughout the 
home through high fidelity speakers placed strategi- 
cally in different zones throughout the home. The selec- 
tions include the output from the compact disk, from a 55 
stereo receiver, from the VCR, from a television and 
also a function to turn the distributed audio off. 

FIG. 3/ shows, a distributed audio sub-menu touch- 
screen, which is displayed after an audio source is se- 
lected in FIG. 3^ or after an intervening menu for a 60 
large house offers floor selections, and allows the user, 
by touching the appropriate portions of the particular 
level of the house ^own as a floor plan on the touch- 
screen, to cause the audio output from the selected 
source from the touchscreen menu of FIG. 3e to be 65 
played through the speakers present in the particular 
rooms selected by the user, llie touchscreen menu of 
FIG. 3/ also provides two function boxes, marked as 



upper level and ground level, so that the other levels of 
the home can be selected to be shown on the screen. In 
that manner, the desired rooms on all floors of the home 
may be included to play the output from the audio 
source chosen by the user. Obviously, other graphics 
designs are possible. 

FIG. 3g shows the security management sub-menu 
touchscreen, which is accessed from the main menu 
touchscreen of FIG. 3a. In addition to selecting one of 
the plurality of levels of the home as a function, which 
would then display, for example, the entry level sub- 
menu touchscreen of FIG. Zh, the user may also select 
to view the output of video cameras distributed outside 
or throughout the home, for example, a video camera at 
the front door or in the garage. The video output from 
those cameras is displayed on the monitor to which the 
touchscreen is attached. It replaces the security man- 
agement sub-menu with the exception of the back and 
quit function boxes. 

From the touchscreen shown in FIG. 3^, a security 
event log function may also be selected, the screen of 
which is shown in FIG. 31. Also, the air conditioning or 
heating blowers may be reenabled by the user after they 
have been automatically shut down by the automation 
system as the result of a smoke or fire detection by the 
security system. 

FIGS. 3A and 3/ show the sub-menu touchscreens 
which appear when the entry level function is selected 
from the security management sub-menu touchscreen 
FIG. 3g. Referring fu^t to FIG. 3A» it can be seen that . 
the screen is dominated by a floor plan schematic of the 
entry level section of the home. By touching the appro- 
priate secured zone, the user can arm the security sys- 
tem, or disable a particular zone, depending upon the 
current state of the security system and the selected 
zone. As shown in FIG. Sh, the user has touched the 
zone 14 which is the right living room window. The 
status information at the bottom of the screen, under the 
zone and name of the zone information, indicates that 
the zone is currently enabled. The system status infor- 
mation to the left of the floor layout indicates that the 
system is ready to arm. In a block above the system 
status information, a zone key is provided. 

The sub-menu touchscreen shown in FIG. 3/ is simi- 
lar to the touchscreen shown in FIG. 3A, with the ex- 
ception that the status of zone 14 is now disabled. Thus, 
the function box provided on the right hand portion of 
the screen is one which would allow that zone to be 
enabled, instead of aUowing the zone to be disabled, as 
shown in FIG. 3h. In the upper left hand comer of the 
floor layout of FIG. 3i it can be seen that the user has 
also disabled another window. Thus, the screen graphi- 
cally shows the present sutus of all security zones on 
the system. In order to arm the security system, the user 
touches the "ARM SYSTEM" box on FIG. 3j and the 
touchscreen of FIG. 3/ appears. 

FIG. 3y shows the touchscreen keypad which may be 
utilized by the user to enter a password or pass code. A 
password is used by the system in order to prevent 
tmauthorized control of certain functions, for example, 
the security function of the instant home automation 
system. After the user has entered his security pass 
code, the screen of FIG. Zk appears. 

FIG. 3k shows a sub-menu touchscreen similar to 
that shown in FIGS. 3/ and 3^, except that the status of 
the security system is now armed. Thus, in the lower 
left portion of the screen, the status "armed" is shown in 
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place of the "disanned" status shown in FIGS. 3/ and 
3A. 

FIG. 3/ shows the security event log sub-menu touch- 
screen which can be reached from the security sub- 
menu touchscreen of FIG. 3g. The touchscreen shown 3 
in FIG. 3/ provides the usual back and quit functions 
and, in addition, provides for scroll up, scroll down and 
clear log functions. In the center of the screen, data is 
shown relating to prior intrusions in each of the zones. 

FIG. 3m shows the lighting mood sub-menu touch- 10 
screen which is reached from the main menu touch- 
screen 3a. Each of the available functions, namely early 
morning, normal day, normal evening, day j>arty, eve- 
ning party, romantic evening and night lights set prede- 
termined lights within the home to predetermined set- 15 
tings which are stored as data files on the hard disk of 
the home automation system and are called up merely 
by the user touching the appropriate function box on 
the lighting mood submenu touchscreen. 

FIG. 3n shows the information sub-menu touch- 20 
screen which may be accessed directly from the main 
menu touchscreen shown in FIG. 3a The available 
functions cause the telephone system in the home to dial 
a telephone number stored in a data file which accesses 
a remote data base. Once the data base is reached, the 25 
information, for example, in the case of weather infor- 
mation, is disseminated to the user either through spo- 
ken report by means of the high fideh'ty speakers placed 
adjacent to the touchscreen currently in use, or through 
a high resolution graphics display created from digital 30 
data received from the remote data base. Thus, the data 
can be presented in either audio or visual format. . 

The software to schedule a device using a graphics 
based system as described above, involves using the 
graphics capabilities of the display generator 54 in a 35 
way to produce pleasing picture elements that can be 
moved under cursor control. The result will be both 
visually pleasing and easy to understand and use. FIG. 
9 shows a software flow chart of a preferred implemen- 
tation of this type of system. Obviously, the software 40 
can be implemented in a variety of ways, depending 
upon the specific screen design. 

Referring now to FIG. 9, from the main program 900, 
the user selects the scheduling screen options subrou- 
tine 902 and then selects the option desired. The sched- 45 
uling screen options subroutine is accessed using the 
system configuration function from the main menu 
shown in FIG. 3a. The program then determines 
whether the option selected is the start date 904 and, if 
it is, displays the graphics for the start date and accepts 50 
that input 906. Once the user makes his selection for the 
start date, the program returns to A, displays the sched- 
uling screen and options 902 and the user then picks the 
next option to be scheduled. In that manner, the user is 
led through a sequence of steps which result in the 35 
display, input and acceptance of the scheduling data for 
the selected device. If the option selected is not the 
option start date, then the program determmes whether 
the option selected is the start time 908. If it is, then the 
graphics for the start time and acceptance of the start 60 
input are displayed 910 and the user is able to enter the 
start time. The program then returns to A, displays the 
scheduling screen and options 902 and waits for the user 
to make another selection. 

If the option selected at 902 is not the start date or 63 
start time, the program then determines whether the 
option is the stop or finish date 912. If it is, then the 
graphics for the stop date and acceptance of the stop 



date input are displayed 914. Again, the user selects the 
stop date, requests that the stop date input be accepted 
and the program returns to A displays the scheduling 
screen and options 902. 

If the option selected at 902 is not the start date, start 
time or stop date, then the program queries as to 
whether the option selected is the stop time 916. If it is, 
then the display graphics for the stop time and accep- 
tance of the stop time input 918 are shown on the video 
display monitor 56. The user selects, using the touch- 
screen 16, the stop time and the program returns to A to 
display the scheduling screen and options 902. In the 
event that the option chosen by the user is not a start or 
stop option, but it is an option which indicates that the 
scheduling has been completed, the program stores the 
schedule and returns to the mam program 920. 

As previously described, this entire sequence can be 
accompanied by verbal prompts, generated by the voice 
recognition and speech synthesis circuitry 58, to prompt 
the user through the scheduling scheme. Many different 
types of events can be scheduled in this manner, such as 
a lighting system event, control of the audio/video 
entertainment system, the energy management system, 
individual appliances and the configuration of the sys- 
tem itself. Thus, system features and functions can also 
be scheduled by the user, such as spoken alerts, pass 
codes, etc. 

FIGS. 10a through lOg show the touchscreens which 
may be displayed in order to enable the user to schedule 
events. For example, FIG. 10a shows a monthly calen- 
dar but also provides function boxes which, upon being 
touched, display either the month before or the month 
following the currently displayed month. The user se- 
lects a day of the month merely by touching that date 
on the screen. Other ftmction boxes provide for a 
weekly cycle selection, copying the time selected, cut- 
ting and saving a schedule for use with another function 
or canceling the date. 

FIG. lOb shows the particular mode which the user is 
currently setting and shows a sunmiary of the settings 
for that mode. For example, as shown in FIG. 10*, the 
information indicates that the system will go into the 
vacation mode on May 24 from 4:30 a.m. to 6:30 a.m. 
and then from 6:35 p.m. to 8:00 p.m. 

In the example shown, the appropriate portion of 
FIG. 10c has been touched by the user to indicate that 
the vacation mode should start at 6:35 p.m. and FIG. 
lOd has been touched by the user in the appropriate 
place to indicate that the vacation mode shoiUd finish at 
8.O0 p.m. 

FIGS. lOe and 10/show alternative methods for uti- 
lizing the touchscreen of the present home automation 
system for setting the start and fmish time. As indicated 
in FIG. 10^ the vacation mode is to run from May 20 to 
May 24. FIG. Ide shows that the vacation mode is to 
start at 1:17 p.m. on May 22 and continue untO 3:22 p.m. 
on Thursday, May 24. By providing the touchscreen 
shown in FIG. 10, or various combinations thereof, the 
user may readily schedule the desired events merely by 
touching the appropriate places on the touchscreen of 
the present home automation system. 

INFORMATION RETRIEVAL 

The present home automation system provides an 
integration of tailored information and communication 
services in a home automation system environment with 
high resolution color graphics displays. Since the sys- 
tem disclosed herein can be connected to the outside 
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world by both telephone and modem, it provides the 
capacity for remote communications and information 
retrieval. 

As shown in FIG. 1, there are a plurality of ways in 
which the home automation system disclosed herein can S 
be connected to sources of external information. Three 
of these are specifically shown as the telephone system 
62, which is connected to the standard telephone net- 
work (not shown), the Ethernet network interface 50 
and the multiple external terminals 52, and a network 10 
file server 51 which together form a local area network, 
and the modem 42 which may be connected to a plural- 
ity of remote data bases 44. Each of those modules is 
ultimately connected by means of the data bus 12 to the 
central processor 10. 15 

Utilizing, for example, the proper telephone interface 
60, and a speech processor 58, the present home auto- 
mation system can make and answer audio telephone 
calls. Tim means that the home automation system can 
be directed from a touchscreen 16 to make telephone 20 
calls to information services which provide audio infor- 
mation. One such application is to retrieve single mes- 
sage recordings such as daily weather reports, financial 
reports or sports reports. An additional application is to 
ax^cess voice "data bases*' such as individual stock prices 25 
or checking account balances, which requires that the 
home automation system send touch tone signals to 
retrieve the proper information. Alternatively, the in- 
formation can be requested and retrieved in a digital 
format from a wide variety of available data bases utiliz- 30 
ing the modem 42. The same is true when using a con- 
nection to a local area network that could provide local 
community information or, through a gateway, access 
to remote information. 

The present invention, by its combination of informa- 35 
tion retrievable with high resolution color graphics, 
provides advantages both in requesting and in display- 
ing information. For example, when requesting infor- 
mation, the displays can provide a well organized menu 
of options so that, with a single touch, the desired infor- 40 
mation can be retrieved. For displaying information, the 
color graphics capability allows the received informa- 
tion to be reformatted into highly readable displays or 
even into charts and graphs. In the future, as the tech- 
nology used by on-line information services improves 45 
and additional services become available, pictorial or 
other information will be able to be retrieved and dis- 
played by the home automation system, as well. 

The foregoing types of information can also be re- 
quested with any wired or wireless cursor control or 50 
general input device, such as a mouse, touchscreen 16, 
light pen, joy stick, keyboard 18 or voice recognition 
system. 

MULTIDIMENSIONAL INTERACTION AND 55 
CONTROL 

The present home automation system provides the 
capability to the user to interact with and control a 
home in a variety of modes in order to increase both the 
capability and ease of use of the home automation sys- 60 
tern. For example, the present home automation system 
incorporates high resolution color graphics displays 
that are essentiaUy menu oriented, partially floor plan 
oriented, and partially mood oriented. Each type of 
display has advantages in different control situations. 65 
Hoor plans are best used when direct control options 
are required for many specific devices. The present 
home automation system utilizes floor plans to activate 



and deactivate individual security zones throughout the 
house by touching the location of that zone on a floor 
plan, using a touchscreen 16. 

The present invention also, however, provides for 
•*mood*' control. Moods are macroinstructions of multi- 
ple commands that allow single commands to set an 
entire living environment in the home, such as all the 
lighting and music for an evening party. On the other 
hand, menus offer the best means to select options that 
are not necessarily related to a location in the home, 
such as a conunand to play a< particular compact disk, 
request a certain type of remote information, or to 
switch to another sub^menu of options. The system is 
provided with the ability to use the floor plan mode to 
create the moods to be used in the menu of moods. 
Thus, the floor plan displays create and define lists of 
tasks to be performed in each particular mood. 

The present home automation system, by reason of 
utilizing an AT compatible bus, allows the use of a wide 
variety of video options. For example, by use of certain 
display driver cards, a range of video resolutions from 
hi^ resolution displays for use with detailed floor plans 
to lower resolution menus can be shown. That provides 
for faster loading of the information into the central 
processor and out to the monitor 56. Also, by using a 
monitor that can show both standard video as well as 
the high resolution display graphics, the display screens 
56 can double as televisions or security monitors. 

This function b accomplished by the use of a multi- 
resolution video display driver card such as the VGA 
VIP card or the VGA Wonder card, available from a 
ATI. It also uses as the primary video monitor, the 
Mitsubishi AUM-1381A which can display either com- 
puter-generated EGAA^GA video or standard NTSC 
television video. 

INTERFACE TO A MULTI-ZONE SECURITY 
SYSTEM 

Another capability of the present home automation 
system is its ability to interface to multi-zone security 
systems and to allow user control of such security sys- 
tems by means of interactive touchscreens 16 and other 
alternative control interfaces. The present home auto- 
mation system is designed to interface to and take ad- 
vantage of the advanced features offered by sophisti- 
cated security systems Thus, the home automation sys- 
tem is designed to operate with an external security 
system, which provides several advantages. 

Hrsl, the present system can be performing control 
tasks while the security system is calling the alarm ser- 
vice. One of those control tasks can be contacting some- 
one else by telephone as a backup to the alarm service. 
Another advantage of being designed to operate with 
an external security system is that the present home 
automation system can monitor the performance of the 
security system and can keep a separate log of all secu- 
rity events. Still another advantage is in the use and 
power of the interaction by the user with the security 
system by means of a hig^ resolution color graphics 
display and cursor control devices. Such an interaction 
allows the user interface to be greatly improved and 
simplified over those previously known. Further, the 
home automation system, since it is designed to operate 
with external security systems, can be utilized with the 
very latest security systems and can take advantage of 
improvements and updates to such security systems. 

Another advantage of integrating a security system 
into the present home automation system is the ability to 
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provide information to the user while announcing the The foregoing function of the system is performed in 
alarm. The system can speak the location of the intruder software by calling a speech subroutine as each new 
or fire whUe displaying that location on a floor plan graphics screen is displayed. The speech information is 
screen of the home. It also turns on all lights in the stored in the extended RAM memory of the processor 
house to light exit routes and shut down heating system 5 10. which has been previously described. The home 
blowers to minimize smoke circulation in the house. automation system utilizes the speech processor 58 and 
A preferred external security system to which the a remote spwiker 66 located at each touchscreen loca- 
present home automation system may be interfaced is tion in order to provide the spoken cues to the user 
the Silent Knight 4000 security system, available from while he is using the touchscreen. 
Silent Knight. That home security system is shown as ID SOFTWARE SYSTEM 

element 3S in FIG. 1. It is Interfaced to the central 

processor 10 by means of a custom RS-232-lo-security As has been described herein, the present expandable 
system translator 36, which in turn is connected to the home automation system is a distributed processor sys- 
serial interface Ma and then through the AT bus 12 to tem, in which a central processor 10 is integrated with 
the central processor 10. The interface or translator 36 15 a collection of secondary processors associated with the 
is a microprocessor-based translator and controller. various interface elements. The disclosed home automa- 

In addition to connecting the home automation sys- tion system is preferably arranged with a star topology, 
tem to the security system bus, it listens to and interprets in which the secondary processors serve to relay infor- 
all data instructions generated by the security system on mation to the central processor, or to translate com- 
its bus, and translates commands from the home auto- 20 mands received from the central processor into com- 
mation system to the security systems* , data protocol so mands that their dedicated devices can understand. As 
that the security system will receive the commands and has been described, the secondary processors manage 
respond appropriately. Besides allowing the security the following sujbsystems: voice recognition and voice 
system 38 to receive and respond to commands from the synthesis; telephone communication; touchscreen corn- 
central processor 10, which are ultimately generated by 25 munication; hand-held remote control unit communica- 
the user of the system, the home automation system is tion; input and output control and monitoring; security 
also able to record security events on a real time basis. and fire system safety and monitoring; and other op- 

As shown in FIG. 2, a four wire bus is utilized by the tional intelligent subsystems, such as lighting, audio and 
Silent Knight home security system 38, two wires for video and HVAC. 

power and the other two for data transmission. The 30 While the secondary processors generally run stan- 
interface translator 36 connects to all four wires of that dard software available from the interface manufac- 
bus. Data originating in the security system 38 is first turer, the instructions for running the central processor, 
level shifted from a data stream between +12 volts and however, have not been described and will thus be 
ground to a data stream between -t- 12 volts and — 12 described in this section of the application. These in- 
volts and then passed to an input bus of a Z80-based 35 structions arc termed the Home Environment Auto- 
microcomputer. Under control of an EPROM resident mated Real-time Transactor or HEART. The HEART 
program, the Z80 microprocessor translates data and instructiohs have been created in a standardized and 
instructions appearing on the bus. Such data and in- modular form such that the instructions may be config- 
structions exit the translator 36 in RS-232 format, from ured for each different home environment, as the user 
which they ultimately reach the central processor 10. 40 desires. The HEART instructions consist of six major 
When the home automation system gives a command to portions, which are described hereafter. Those portions 
the security system, it passes through the Z80 micro- are (1) the Subsystem Interrupt Handler; (2) the Internal 
processor, is converted to the proper format and timing Scheduled Interrupt Server (ISIS); (3) the Polling 
for the security bus, passes through another level shifter Loop; (4) the Tasks; (5) the Device I>rivers; and (6) the 
and is then converted to the H- 12 and ground format 45 Support Environment. 

used by the security system 38. Briefly, the Subsystem Interrupt Handler operates in 

eT^vT-vT xMTTca A i-cc A c r^iT^c background in order to receive itaessages from the 

SPOKEN MESSAGES AS CUES secondary processors and to build received message 

The present home automation system also provides queues. The Internal Scheduled Interrupt Server han- 
for the use of spoken messages as cues for touchscreen 50 dies the scheduling of timed events. The Polling Loop 
use, providing a great level of ease of use for non-tech- monitors the message queues and the ISIS and calls the 
nical users of the system. The present home automation appropriate Task. The Tasks respond to the messages 
system provides spoken cues to help guide the user from the secondary processors. The Device Drivers 
when using touchscreens to the next step. The present translate internal commands into the instructions neces- 
home automation system incorporates hardware and 55 sary to execute the desired function. The Support Envi- 
soft ware to support the playback of high-ftdelity, digi- ronment consists of the additional task-switching appa- 
tally recorded messages to the user. For example, when ratus and internal tools that allow the various subsys- 
the user touches one of the touchscreens 16 to request tems to ftinction in a cohesive and multitasking manner. 

rSltV^.Se'^SSc"'^^^ ^'TiS^^^ « ™ SUBSYSTEM INTCRRUPT HANDLER 

screen graphic is displayed. For example, on the main A flow chart of the functions performed by the Sub- 
menu, the user may select the ^Security Management" system Interrupt Handler is shown in FIG. 4^. All of 
option. If the user lived in a large house, his next step the tasks and devices based on a serial line, such as the 
might be to select a floor or area of the house he wishes touchscreen, hand-held remote control unit and secu- 
to control. By speaking the phrase, please select your 65 rity system, are supported by the interrupt handler, 
floor" upon displaying a floor selection menu, the user running in the backgrotmd. When a byte of data is re- 
is assisted in continuously understanding the next func- cdved at any serial port, an interrupt request line is 
tion to be performed. raised, which causes the interrupt hsundler to execute. 
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The inierrupi handler determines which serial port 8. Master touchscreen (1) 

caused the interrupt, copies the new dau into that port's 9. Secondary touchscreen (2) and other touchscreens 

queue and then increments that queue's pointer. The 10. Serial waU switch bus 

Polling Loop, which is shown in HGS. 5fl and Sb, 11. Telephone control interface 

monitors the queue pointer and jumps to the queue's 5 12. Other intelligent subsystem mterfaces 

task when the queue pointer moves. A flow chart showing the functions of the PolUng 

As shown in FIG. 4A. when the interrupt request line Loop is shown in HGS. Sa-Sa From the top. of the 
is raised the interrupt handler begins to execute. Ini- Polling Loop, a determination is made as to whether 
tially, the interrupt handler jumps to the interrupt han- new security or fire 1 data 502 has bera received from 
dler routine 300. The interrupt handler routine then 10 the serial line 1, the serial line queue which corresponds 
determines, by asking each serial chip, which serial Une to one line from the serial interface 14 If new data h^ 
generated the interrupt 302 which caused the interrupt been received, the data is mierpreted 504 and. if found 
request line t be raised. Upon determining which serial to be valid data, is sent for execution to a task 1 state 
line generated the interrupt, the appropriate data byte is decoder 506 which selects which section of code or 
puUed from the identified serial lines, RX register 304 15 state in the Task will process the new data. The pro- 
Md is placed in the appropriate serial queue 306. That gram then returns to the top of the pollmg loop 500. If 
queue's pointer is then mcremented 308. If the queue the new data is determined to be mvalid data at the data 
pointer is greater than the queue size, then the. queue interpreter step 504, then the program jumps immedi- 
pointer is wrapped around. The interrupt handler rou- ately back to the top of the Pollmg Loop 5(M). All other 
tine then ends and returns to the normal program 310. 20 tasks (2 through 9) follow the same format of data 

interpreter" and "state decoder." Further discussion of 

THE INTERNAL SCHEDULED INTERRUPT specific Tasks follows. 

SERVER QSIS) If security and fire 1 dau has been input, then 

The Internal Scheduled Interrupt Server or ISIS is a a determination is made as to whether there is new 
tool available to any task, device driver or ISIS event. 25 security and fire 2 data 508. If there is. then Task 2 is 
It allows a routine to schedule a software event to occur executed 510 based upon that data and then the program 
any time within the next 24-hours. The ISIS load event returns to the top of the polling loop 500. If no new data 
routine is called with an event time, event number and is in the serial queue 2 at the step for determinmg 
optional data. The new event is inserted into the ISIS whether new security/fire 2 data has been mput 508, 
queue in accordance with its event time. The top of the 30 then the program next detenmnes whether the current 
ISIS queue is checked in the Polling Loop, and, if the time in the Internal Scheduled Interrupt Server event 
current system time is equal to or greater than the time queue 512 is greater than the next event m the queue 
ofthe top event in the queue, that ISIS event is removed 514. If it is, the Internal Scheduled Interrupt Server 
from the queue and then executed. subroutine is called 400. After the execution of the ISIS 

The types of ISIS events available are monitoring. 35 subroutine, the program returns to the top of the Polhng 
scheduled subsystem events, wait for acknowledgment. Loop 500. 

wait for data, and support environment events. If there is no internally scheduled event or interrupt 

FIG 4a shows a flow chart of the operation of the to be executed at the current time 514, then a determina- 
intemai scheduled interrupt server. As previously de- tion at step 516 is then made as to whetiier there is any 
scribed, when tiiis tool is called from the PoUing Loop. 40 new input/output controUer data at the serial queue 3 at 
the instruction at tiie top of the queue whose time has point 518. If tiiere is, then Task 3 is executed 520 and the 
been passed is to be executed 400. Examples of the input/output controller subroutine is executed. The 
various types of ISIS events are described later in con- program then returns to the top ofthe Pollmg loop 500. 
nection with FIGS. 13fl through ISd If there is no input/output controUer daU at step 516, 

45 a determination is then made as to whether a key has 
THE POLLING LCX)P pressed 522 on the keyboard 524. If a key has been 

The PoUing Loop forms a part of the virtual mul- pressed, then certain diagnostic routines 526 may be 
titasking environment utiUzed by the instant home auto- caUed and then the program returns to the top of the 
mation system, since all tasks are executed incremen- PoUing Loop 500. Alternatively, although not shown m 
taUy depending upon inputs noticed by the Polling 50 HG. Sc. once a determination tiiat a key has been 
Loop. The Polling Loop processes and checks the vari- pressed 522 is made, another determination as to 
ous inputs in priority order from high to low. Whenever whetiier a diagnostic routine has been called may be 
an input presents data to be processed, tiie PoUing Loop made. If a diagnostic routine has not been called, then 
executes an immediate jump to that input's task. When the system wiU interpret the keyboard input to be m- 
the task has completed executing its current state using 55 structions or data and wiU execute those instructions or 
the new data, it immediately jumps back to the top of data accordingly. 

the PolUng Loop. However, some tasks may immedi- If the determination at step 522 of whether a key has 
ately execute their next state without returning to the been pressed is negative, then a determination is made as 
PoUing Loop if more data is available. to whether data has been received from touchscreen 1 

In its preferred embodiment, tiie Polling Loop moni- 60 at point 528. If Uiere is touchscreen data on the queue 
tors tiie foUowing input sources in order of priority: 530 from serial Une 4, then tiie touchscreen Task 4 is 

1. Fire/security 1 executed 532. The program then jumps to the touch- 

2. Fire/security 2 (optional) screen 1 or master touchscreen subroutine, a flow chart 
3" Internal scheduled interrupt server of which is shown in FIG. 6. The program then returns 

4 I/O conuoUer 65 to the top of the PolUng Loop 500. 

5 Keyboard The second haWoftiie PoUing Loop is shown m FIG. 

6 Voice recognition board 0 Sb. As shown in -tiiat figure, a determination is tiien 
7. Voice recognition board 1 (optional) made as to whether Uiere is new touchscreen 2 data 534 
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by examining the serial queue 5 connected to touch- If the current state is detenmined to not be the neutral 

screen 1, If there is data present, then the program exe- state, then the system checks voice recognuion to deter- 

cutes Task 5 for the touchscreen 2 536 and jumps to the mine whether a template number has been returned 

touchscreen 2 subroutine, which is simUar to the touch- from the currently active voice recognition board 580 at 

screen 1 subroutine shown in FIG. 6. Note that addi- 5 step 578. After steps 570 and 574, if it is determmed that 

tional secbndary touchscreen tasks may be inserted in a template number has been returned, the program pro- 

the Polling Loop at this point. ceeds to again deterncdne whether a temp^te number 

If no new touchscreen 2 data is present at the serial has been returned at 578. If a template number h^been 
Queue 5. a determination is then made as to whether a returned, then the recognition score is displayed 582 
particular function switch, for example, a serial waU 10 and the simulated touchjroordinates are set to the mid- 
^tch 13 has inputted data on serial line 26 to the serial die of the screen 584. The system then jumps to the 
aueue 6. If new function switch 1 data is present 538. master touchscreen immediate response module 586. 
Sien Task 6. the task associated with that particular In the event that no template number has been re- 
function switch 540 is executed and the program then turned at step 578 or if the voice recoguiUon is deter- 
retums to the top of the Polling Loop 500. If no new 15 mined to not have been enabled, the system moves to 
function switch 1 data is present, then a determination is the touchscreen 1 queue 558. 
then made as to whether new function switch 2 data is TUB TASKS 
present 542 on serial line 7 in serial queue 7. If new data . • r 
\s present, then Task 7. which corresponds to the func- A Task, as that phrase is used herem. is a software 
tion switch 2, is executed 544 and the program returns 20 module that supports one specific piece of hardware 
to the top of the Polling Loop 500. Note that additional within the disclosed home automation system. The 
function switch tasks may be inserted in the Polling communication protocols of both the tasks and the 
Loop at this point. secondary processors are designed m such a manner to 

If no new data from function switch 2 has been input allow the Tasks to run in a high-level sunulaiion of a 
on serial line 7, the program then determines whether 25 multitasking system. Each task is brokwi up mto sub- 
an incoming phone call is being received 546 from the tasks, which are in turn themselves divided mto single 
telephone interface 60. If an incoming telephone call is --states.^ Each state is designed to execute m one second 
beinc received, then Task 8. answering the phone, is or less, and. on completion, returns to the top of the 
executed 548. If no incoming telephone call is being Polling Loop. Transitions between states are triggered 
received, then the program determines whether the 30 by the interrupting events recognized by the PoUmg 
phone call is active and touch tones are recognized 550. Loop. , . i. 
If an affirmative determination is made, then Task 8 is The various devices controlled and momtored by the 
executed, the incoming touch tones are decoded 552 home automation system described herem are assigned 
and the program then returns to the top of the Polling to tasks in the following manner. Every secondary pro- 
Loop 500. If no touch tones are recognized, the pro- 35 cessor which communicates with the central processor 
gram returns to the top of the polling loop 500. over a serial line is assigned its own task. Devices on the 

FIG. 5r is a flow chart showing the voice recognition internal AT bus 12, which use an Interrupt Request 

portion of the Polling Loop. After it has been deter- Line (IRQ Line) may be assigned theu- own tasks. TTie 

mined that a key of the keyboard 524 has not been assignment of tasks to actual hardware occasionally 

pressed, a determination is made as to whether the voice 40 demands that more than one device be tied to one t^k. 

recognition circuitry is enabled 554. If that determina- Furthermore, the task number assigned descnbes that 

tion is affirmative, then a determination is made as to task's priority. That is, task 1 is the highest priority task 

whether a telephone call is currently active 556. If a in the system. The task assignments for the preferred 

telephone call is currently active, then the system skips configuration are set forth below, 

to the touchscreen 1 queue at step 558. 45 Task 1: Fire and security interface ^ ^ . 

If. at step 556, there is no active telephone call, then Task 2: Second fire and secunty mterface (optional) 

a determination is made as to whether the current active Task 3: Input and output controller interface 

board is the speech board 560 (the board also used to Task 4: Master touchscreen and voice recognition 

generate spoken responses). If the current active board boards 

is the speech board, then a determination is made as to 50 Task 5: Secondary touchscreen (optional) 
whether the speech board is in the talkmg mode 562. If Task 6: Serial wall switch 1 bus 
the speech board is in the talking mode, then the Ini- Task 7: Serial Wall Switch 2 bus (optional) 
trccognition subroutine is called 564. The system then Task 8: Telephone control interface 
determines whether the current Master Touchscreen Task 9: Other intelligent subsystem mterfaces 
state is the Neutral State 566. 55 AU tasks are structured in approximately the same 
If a determination at steps 560 or 562 is negative, the manner Mid consist of the foUowing pieces: (J) data 
system then proceeds to determine whether the current validation routine: (2) immediate response routines: (3) 
state is the Neutral State at step 566. If the current state neutral state: and (4) contextual states. Each of those 
IS the neutral sute, then the recognition on board 0 is components is described further below, 
checked 568 and then a determination is made as to 60 When a Task receives data, it must first evaluate it. 
whether a template number has been returned 570. If no The data validation routine (DVR) consists of a se- 
template number has been returned at step 570, then the quence of states, in which each state evaluates one byte 
recognition on board 1 is checked at 572. A determina- of data. If the byte is rejected, the state resets the DVR. 
tion is again made as to whether a template number has If the byte is not rejected, the state increments the state 
been returned 574. If no template number has been 65 variable to the next DVR state and returns to tiie Pol- 
returned, then the recognition on board 2 is checked ling Loop, thus allowing the next byte of data to cause 
576 and tiie program again determines whether a tem- the next DVR state to execute. The communication 
plate number has been returned 578. protocol for tiiat task's device is encoded mto the DVR. 
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TABLE 1 -continued 



Example or a Specific Task Description ■ 
Task I: the Ftre/Security Interface 



10 



in order to form the criteria by which the DVR wiU 
accept or reject data. The DVR also simultaneously 
decodes the data to provide necessary information to 
the remainder of the task. 

Immediate response routines— when a valid complete 
transmission has been received, most Tasks will per- 
form some immediate action, such as acknowledging 
the transmission or displaying a cursor on the touch- 
screen. These immediate responses are performed re- 
gardless of the current state of the Task. Once the re- 
sponses have been performed, the software will immedi- 
ately jump to either the Neutral State or to the appro- 
priate Contextual State. 

Neutral State— The Neutral State is a catchall state 
which is executed when the Task has not been placed in 
any other (contextual) state. Most unpredictable and 
unexpected events, such as a fire alarm, are processed 
by a Neutral State. The Neutral State also serves to 

ignore garbled or meaningless transmissions. j^\o Shum'zi)^tetaie of Master Touchscreen Task 

Contextual States— Contextual States form the ma- ^ 
jority of the intelligence of the instant home automation execute Neutral State without changing Conicxtiial State 
system. When the data reaches the current Contextual «jd if ^ . 

SUte. it is acted upon in the context of that current Waitmf^ for interface to rr^um zone map 

state. For example, data indicating that the fourth touch 

box of the touchscreen menu has been touched when ^ ^^^^ ^ requested a zone map 

else if command is rejected password 
resuh = rejected password 
jump to subsiate that requested a zone map 
else 

execute Neutral State without changing Contextual State 



Touchscreen Task 

else if command is $>-stem already disarmed 
result — already disarmed 

jump to Disarm Security System subsiate of Master 
Touchscreen Task 

else if command is password rejected 
result — password rejected 

jump to Disarm Security System substate of Master 
Touchscreen Task 
rise 

execute Neutral Slate without changing Contextual Stote 
end if 

Waiting for a response to shunt zone command: 
if command is acknowledgment then 
result » good 

jump to Shunt Zone substate of Master Touchscreen Task 
dse if command b rejected password then 
fcsoh s rejected password 



the touchscreen task is in the "lighting" mood Contex- 
tual State would be evaluated on that basis, resulting in 
some action, such as calling the lighting mood device 
driver to set lighting mood number 4. Some Contextual 
States may jump to the Neutral State if the data does not ^ 



^^At,^^ au«^ J ••""J' - — 

make sense to the current Contextual State. However, 
when that occurs, the Task's state variable will not 
change, and will still direct future data to the same 
Contextual State. 

An example of a specific task description is given in 
Table 1 for Task 1, the Fire and Security Interface. 
TABLE 1 

Example of a Specific Task Description - 
Task 1: the Fire/Security Interface 

Data Validation: 

The data validation consists of tfie following states: 
1st Synch byte: Must be equal to 52 or data is rejected 
2nd S>Tich byte: Must be equal to 126 or transmission is 
rejected 

Length byte: Receive length of message 

Command bjie: Receive command byte 
Data bytes: Receive appropriate number of data bytes, if 

any 

Checksum byte: Must be equal to stmi of all bytes in message 

MOD 256 or transmission is rejected 
Immediate Response: 

Call security device driver to send acknowledgment of 
receipt of message. 
Neutral State: 

If command is Hre Alarm, then 
set Master Touchscreen task to Fire Alarm State, 
jump to Fire Alarm State in Master Touchscreen Task, 
end if 

If command is Sectnity Alarm, then 
set Master Touchscreen task to Security Intrusion State, 
jump to Security Intmaon State ro Master Touchscreen Task 
end if 

If command b any other type of alarm, then 
call lighting moods device driver to turn on all lights 
end if 

ignore all other co m ma n ds 
return to Polling Loop 
Contextual States: 

Waiting for response to disarm system request: 
if command is acknowledgment then 
result = good 

jump to Disarm Security System subsiate of Master 



end if 

Waiting for response to arm system request: 
if command is acknowledgment then 
result B good 

jump to Arm Security System substate of Master 
35 Touchscreen Task 

else if command is system already armed 
result B= already armed 

jtrajp to Arm Security System substate of Master 
Touchscreen Task 
else if command is system not ready 
40 result ^ not ready 

jump to Arm Security System subsiate of Master 
Touchscreen Task 

dse if command is password rejected 
resuh = password rejected 
jump to Arm Security System substate of Master 
45 Touchscreen Task 
else 

execute Neutral State without changing Contextual State 
end if 

A flow chart showing the function of Task 4, the 
touchscreen 1 or Master Touchscreen Task, is shown in 
FIG. 6. From the top of the Polling Loop 500. when the 
determmation of whether there is new touchscreen 1 
data 528 is in the affirmative, a determination is then 
55 made as to whether that date is valid 600. If the date is 
not valid, then the task is terminated and the program 
returns to the top of the Polling Loop 500. If the date is 
valid, a determination is then made as to whether suffi- 
cient ^te has been received to calculate the touchpoint 
60 602. If insufficient date has been received, then the stete 
variables are set such that further date will be used to 
complete the calculation, the task is terminated, and 
control returns to the top of the Polling Loop 500. 
If, on the other hand, sufficient date to calculate the 
65 touchpoint has been received, then a determination is 
made of whether the touchpoint is greater than 7 pixels 
from a prior touch and. if so, a new cursor is displayed 
604. 
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A detennination is then made as to the state of Task 
4 which causes the program to jump to a sub-task of 
Task 4 corresponding to the function selected by the 
user. For example, the program can jump lo the dor- 
mant screen 608, to the main menu 610, to the lighting 5 
mood menu 612, to the audio/video menu 614, or to 
other menus to control other functions described in this 
application. A different subsystem can be controlled 
from each of those menus. For example, as shown in 
FIG. 6b» the main menu, which is a sub-task of Task 4, 10 
when selected, becomes the current state of Task 4. The 
program then calls the Checktouch subroutine 618 
which is shown in FIG. 11. 

After performing the Checktouch subroutine, the 
main menu subroutine determines whether more touch- 15 
screen data is pending 620. If there is more touchscreen 
data pending, then the program returns to the top of the. 
Polling Loop 500. If there is no more touchscreen data 
pending, then the subroutine removes the rettim to 
dormant screen ISIS event 622, also described later as a 20 
••Screen Timeout" ISIS event, and then, according to 
the last box that had been touched from the touch- 
screen, goes to the appropriate subroutine 624. For 
example, the program can return to the dormant screen 
626. If the last box touched on the touchscreen is the 25 
lighting mood menu, then the program will jump to the 
subroutine which loads the lighting mood menu and sets 
the state of the Task for the lighting mood 628. The 
program then returns to the top of the Polling Loop 
500. 30 

If another box from the main menu touchscreen has 
been selected, such as the audio and video menu, the 
program jumps to the subroutine which loads the audi- 
o/video menu data and then sets the state of the Task 4 
to the audio/video menu 630. The program then returns 33 
to the top of the Polling Loop 500. 

FIG. 6c shows the lighting moods menu subroutine 
which is one of the subtasks of Task 4. From the light- 
ing moods menu 632, the Checktouch subroutine is 
called 618 and then a determination is made as to 40 
whether more touchscreen data is pending 620. If more 
touchscreen data is pending, the program returns to the 
top of the Polling Loop 500. 

If no more touchscreen data is pending then, depend- 
ing upon the box touched on the touchscreen, the pro- 45 
gram jimips to one of a plurality of subroutines 634. For 
example, the program can return to the main menu 636. 
It can return to the dormant screen 626, it can set the 
tighting mood selected by calling the appropriate De- 
vice Driver 638 which then shows a blank lit box 640 50 
and the program then returns to the top of the Polling 
Loop 500. 

Another subtask of Task 4 is the dormant screen, the 
functioning of which is shown in flow chart form in 
FIG. 6d From the dormant screen 642, a determination 55 
is made as to whether more touchscreen data is pending 
620. If more touchscreen data is pending, the program 
jumps to the top of the Polling Loop 500. If no more 
touchscreen data is pending, the program jumps to 
another subroutine 644 which initiates several func- 60 
tions. First, data corresponding to a greeting to be given 
the user is loaded from the processor memory into the 
central processor 10 from which it is sent to the speech 
processor 58 which speaks the greeting over one of the 
remote speakers 66. The main menu graphics and data 65 
are then loaded and the state is set to the main menu. 
The "return to dormant" screen ISIS event, also de- 
scribed later as the **Screen Timeout" ISIS event, is 



then set up to occur in n seconds. The program then 
returns to the top of the Polling Loop 500. 

The functions which are performed when the pro- 
gram returns to the main menu 632, for example, as 
shown in FIG. 6c, are illustrated in flow chart form in 
FIG. 6e After the program branches back to the main 
menu, the main menu subroutine removes the prior 
"return to dormant" interrupt server event 646, sets a 
new "return to dormant screen" ISIS event 648 and 
then loads the main menu graphics and data and sets the 
state of Task 4 to the main menu 650. The program then 
returns to the top of the Polling Loop 500. 

In the event that the return to dormant screen subrou- 
tine is called out from, for example, the lighting moods 
menu as shown in FIG. 6c, the dormant screen subrou- 
tine, shown in flow chart form in FIG. 6/ is imple- 
mented. Upon returning to the dormant screen 632, the 
subroutine removes the prior 'Vetum to dormant" inter- 
rupt server 652 and then loads the dormant screen 
graphics and data and sets the current state to the dor- 
mant screen 654. The subroutine then returns to the top 
of the Polling Loop 500. 

The Checktouch subroutine 618 discussed in connec- 
tion with FIGS. 6* and 6c is shown in FIG. 11. The 
Checktouch subroutine functions to compare the touch- 
point of the user with the touch boxes of the menu 
shown on the touchscreen. From an active menu 700, a 
determination is made as to whether the touchpoint of 
the user is within any of the menu's touch boxes 701. If 
the touchpoint on the touchscreen is within any of the 
boxes on the menu, then the box touched is lit up 706. If 
the touchpoint of the user on the touchscreen is not 
within any of the menu's touch boxes, then, if a box was 
previously lit, it is blanked. After the appropriate box 
has been lit up 706, if a previous box had been lit, then 
that box is blanked. In either event, after a determina- 
tion is made as to whether a box was previously lit and, 
if so, it has been blanked, the program returns to the 
active menu 704 from which it jumped to the Check- 
touch subroutine. 

From the Polling Loop shown in FIG. 5j, if new 
touchscreen 1 data is present 528, then the master 
touchscreen, Task 4, is executed. Initially, a determina- 
tion is made as to whether a new byte is present in the 
master touchscreen queue 708. If a new byte is present 
in the master touchscreen queue, then the program 
determines which of the V-states 710 the Master Touch- 
screen Task or subroutine is currently in and then jumps 
to the entr>' point of the master touchscreen subroutine 
for the particular V-state condition. A V-state, for pur- 
poses of this application, is a state within a data valida- 
tion routine of a Task or subroutine. V-state is also the 
name of a variable used by the state decoders of the 
present system to select which state within a Task or 
subroutine is the current active state. 

Once the appropriate V-sute 710 is selected, the 
program goes to that validation state. Asstmiing the 
current V-state is V-state 0, the program then deter- 
mines whether the bits of the first byte in the master 
touchscreen queue indicate that the new byte is the first 
byte of a four-byte group describing the touchpoint 712. 
If the bits are synchronized, then the system calculates 
the upper half of the touchscreen X-coordinate 714, 
increments the current V-state 716 and then makes a 
determination as to whether more master touchscreen 
data is available 718. If no more master touchscreen 
data is available, then the program returns to the top of 
the PoUing Loop 500. If a determination is made that 
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the bits of the first b>te are not synchronized at step 712. 0. 1 seconds. A determination is then made as to whether 

then the reset synchronization routine is executed at more touchscreen task data is pending 766. If there is 

^20 more touchscreen task data pending, then the system 

If the determination at step 710 is made that the cur- jumps to the top of the Polling Loop 500. If there is no 

rent V-state is V-state 1, or if more data from the master 5 Master Touchscreen Task data currently pending, then 

touchscreen is available at step 718, then the program the system erases the cursor 768 drawn in step 758. and 

jumps to determine whether the bits in the second byte then turns on the speaker in the house closest to the 

from the master touchscreen queue arc synchronized active touchscreen 770. 

722. If they are. then the system calculates the lower The system then makes a determination as to whether 
halfofthe touchscreen X-coordinate and then the entire 10 there is an active phone call 772. If there is, then the 
touchscreen X-coordinates 724. The program then in- message "I'm sorry, but the system is currently respond- 
crements the current V-state 726 and makes a determi- ing to a telephone call" is caused to be sent out over the 
nation as to whether more master touchscreen data is speaker 774 and the program then returns to the dor- 
available 728. If no master touchscreen data is available, mant screen 776. 

then the program returns to the top of the Polling Loop 15 If there is no active phone call at step 772, then the 

500. As described above in connection with V-state 0, if system determines whether the password protection 

the bits of the third byte are not synchronized at step function has been activated 778. If it has. then the pass- 

722, then the program executes the reset synchroniza- word sub-menu is called 780 which requests that the 

tion routine 730. user input the password. If the inputted password is 

In the same manner as described above in connection 20 valid 782, or if password protection is not on. then the 

with V-states 0 and 1, if at step 710 the current V-state stored greeting selected by the user is sent out over the 

is V-state 2 or if at step 728 more master touchscreen speaker at step 784 and the main menu state is initialized 

data is available, a determination is made as to whether 786. If a determination is made that the password input- 

the bits in the third byte are synchronized 732. If they ted by the user is not valid, then the program returns to 

are, then the upper halfofthe touchscreen Y-coordinate 25 the dormant screen 776. 

are calculated at 734, the current V-state is incremented If. as discussed in connection with FIG. 7c the mam 
at 736 and then a determination is made as to whether menu state is initialized at 786, the master touchscreen 
more master touchscreen data is available 738. If there subroutine jumps to the initialization entry point 786 
is no more master touchscreen data available, then the shown on FIG. Id, which is a flow chart of the general 
program returns to the top of the Polling Loop 500. If, >0 Contextual State of the master touchscreen task or sub- 
however, there is more master touchscreen data avail- routine. After the initialization entry point, the system 
able, or if at step 710, the current V-state is V-state 4, sends a '^prompt" phrase over the speaker 788, such as 
then the program determines whether the bits in the "enter your selection", or any other phrase preselected 
fourth bit are synchronized 742. If they are. then the by the user and stored as a daU file on the system hard 
system calculates at 746 the lower half of the touch- 35 drive. The general Contextual State master touchscreen 
screen Y-coordinates and then the touchscreen Y-coor- subroutine then loads the main menu screen and indi- 
dinates themselves. The system then resets the V-state cates the status of the menu options, if necessary 790, 
to the initial V-state 0 at 748 and then the master touch- The speech recognition vocabular>'. if applicable, is 
screen executes an immediate response at 752. then loaded 792 and the system then seu the state to the 

The system operates in the same manner with respect 40 current Contextual State entry point 794 and returns to 

to a determination as to whether the bits in the third and the top of the Polling Loop 500. 

fourth bytes are synchronized as described above in The entry point into the general Contextual State 

connection with steps 712 and 722. If a determination at master touchscreen subroutine from the Polling Loop is 

steps 732 and 742 is negative, that is, that the bits within shown at step 796, also in FIG. Id. The system then 

the bytes are not synchronized, then the reset synchro- 45 calls the Checktouch subroutine shown in FIG. 11 at 

nization routine is executed at 740 and 750, respectively. step 798 and then calls the display ctirsor subroutine 

FIG. lb shows the master touchscreen immediate shown in FIG. 7c, at step 799. A determination is then 
response routine. After executing the master touch- made as to whether one of the boxes on the main menu 
screen immediate response 752, the system converts the has been touched 711. If none of the boxes have been 
touchscreen coordinates calculated in the master touch- 50 touched, then the program returns to the top of the 
screen validation routine to pixel coordmates 754. A PoUing Loop 500. Boxes are defined as areas on the 
determination is then made as to whether the particular screen which are marked off in some way to indicate the 
touch of the touchscreen by the user being analyzed is user should touch there to execute an action. For in- 
greater than 7 pixels away from the last touch of the stance, the current maximum number of boxes a stan- 
touchscreen 756. If an afiirmative determination is 55 dard menu may have is 9. They are Back. Quit, and up 
made, then the prior cursor is erased, a new cursor is to seven other choices. See. for example. FIG. 3fl. 
drawn and the new touch coordinates are stored 758. If one of the boxes has been touched, the program 
As previously described, the cursor may be shown on determines which of the boxes have been selected 713 
the screen as a highlighting or an inverse video function arid jumps to the appropriate point. If the Quit box has 
of the selected menu item on the touchscreen. 60 been touched, then the system returns to the dormant 

If the new touch is less than 7 pixels from the last screen 715. If the Back box has been touched, a dctermi- 

touch or after step 758, the system sets a touch flag 760 nation is made as to whether the system is to go back to 

and then jumps to the Master Touchscreen Task sute the main menu 717. If the answer is afTirmative, then the 

decoder 762. system returns to the main menu 719. 

FIG. 7c shows the Neutral State subroutme of the 65 If the back function selected by the user does not 
Master Touchscreen Task routine. From the Master refer to the main menu, that is, if there are prior sub- 
Touchscreen Task state decoder 762. the system pauses menus between the main menu and the current sub- 
for a predetermined amount of time 764, for example. menu, then the system removes the **screen time out" 
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from the ISIS, inserts a new ♦^screen lime out'* ISIS show a cursor 199, but will not be compared with the 
event and sets the touch flag to Oat step 721. The prior touch boxes on the menu. 

sub-menu or stale is then initialized 723. From the zone map request entry pomt 424. the pro- 

If at sten 713, the box selected, instead of being a gram then receives the type of resuh from the Secun- 
funJtion box such as the Quit or Back ftmction is a menu 5 ty/Fire Task. (Steps 824. 846 m FIG. 8c; or step 1320 m 
selection box, then the program executes the selected FIG. 13c) a * a^a 

action 725 and determines whether the action selected Depending upon the result determined at step 426, 
rSuires branching to a sub-menu 727. If a new su^^ the system may go directly to decode the armed or 
maiu is required, then the system removes the current ready condmons of the secunty system and detenmne 
"screen time out- event from the ISIS and inserts a new 10 that the security system is am«J o^ 
"scr«m time out" ISIS event 729. That causes the next Or, it may reach step 428 either direcUy or after mfonn- 
screen shown on the graphics display monitor to remain ing the user of the response received at step 426^ For 
there mitil the current "screen time out" event is re example, if no response was received at^ep 426 the 
moved from the ISIS. The sub-menu selected by the system speaks the phrase "no response at ^0 andthen 
«Ls^r U^then initialized 731 ^5 moves to step 428. If, on the other hand, an ^'unexpected 

If a negative determination is made at step 727, that is, response" was received at step 426. then the system 
the pro^ b not going to display a new sub-menu, mform the user by speaking the phrase unexpected 
then thec^ent hi^ghted box is blanked 733, thus response" 432. before movmg onto step 428 
re^o W Sie highlgting of the selected box on the After step 428. at step 434, the "please wait' me^ge 
^S«i. Thra^dve touchscreen serial queue is 20 displayed at step 412 is erased from the screen of the 
then cleaned out 735 and the system returns to the top monitor and the sj^em then J^P^J^ i?"*^ 
of the Polling Loop 500. status entry pomt 436 which is described at 303 on FIG. 

The display cursor subroutine called at step 799 of 7/l . i- < j- i 

FIG 7d is shown in FIG. le. After being called from a FIG. Ih shows the entry pomts for the redisplay 
Contextual State of the master touchscreen subroutine 25 security floor plans function 301. the show zones, status 
at step 799, the display cursor subroutine starts at 737 303 and the show secunty system status 305. m addition 
and Aen makes a determination as to whether more to continuing the flow chart from the substate d«:oder 
touchscreen data is available 739. Ifthere is more touch- 438. When the redisplay secunty floor P^an^^l 
screen data available, then the system returns to the top jumped to, the system sets the fade color to black, loads 
of the Polling Loop 500. 30 the floor plan screen accordmg to the current floor 

If no more master touchscreen data is available at number and then sets the fade color to normal 307. The 
739 then the system pauses for a predetermined time, show zones, status 303 entry pomt occurs after step 307. 
such as 0.25 seconds 741. A detennination is then made The program then decodes the zones, status 309 and 
as to whether more touchscreen data is available 743. If marks the zones on the screen as weU as displaying the 
there is no more master touchscreen data available, then 35 status of any open or shunted zones 311. After step 311. 
the cursor is erased at step 745 and a detennination is the system reaches the show secunty system status 
made as to whether a box on the menu has been touched point 305. The security floor plan sub-menu routme of 
747 If more master touchscreen data is available at step the master touchscreen task then shows the aimed or 
743 or if no box has yet been touched at step 747. the ready status on the screen of the monitor 313 and deter- 
system returns to the top of the Polling Loop 500. 40 mines whether the user has previously selected a zone 
If a box has been touched from the touchscreen at 315. If the user had selected a zone at step 315. the 
step 747. then the system highlights the touched box 749 selected zone is marked 317 and the pnnt zone status 
and then returns to the calling subroutine 751. subroutine is caUed at 319. If no zone is select«l at step 

HGS. Ig through In depict a flow chart for the secu- 315, the phrase "no zone selected is pnnted or dis- 
rity floor plan sub-menus for the master touchscreen 45 played on the screen 321. . . „ ^ 

subroutine. At the initialization entry point 410, the user After the print zone status routine is called at step 319 
has selected a security/fire option from the main menu or the phrase "no zone selected' is^prmted at step 321, 
and has selected from the security management menu the system inserts a "screen time-out" ISIS event and 
shown in ¥IG 3g one of the three levels shown of the sets the touch flag to 0. at step 323. The state is then set 
home. The security floor plan sub-menu subroutine then 50 to "analyze floor plan touches" at 325 and then turns to 
stores the selected floor number, loads the selected floor the top of the Polling Loop 500. 
plan screen data, speaks a responsive phrase through the After the current substate is obtamed from the sub- 
beaker and voice synthesizing system, such as "Please state decoder at 438, the system then analyza ^tl>er 
wait for response form the security system" and then any floor plan touches have been made. The Qieck- 
displays a "please wait" message 412. The subroutine 55 touch subroutine of FIG. 11 is first called at 798 and 
then sends a "zone map request" to the security/fu-e then a detennination as to whether a box on the touch- 
interface shown and descn*bed in connection with screen has been touched is made at 71L If no box has 
FIGS 8fl-8c sets the Security/Fire Task to the "zone been touched, then a determination is made as to 
map request" state; and inserts a "security response * whether a new zone has been touched at 440. If a new 
expected" ISIS event into the ISIS queue 414. The 60 zone has been touched, then a determination is made as 
Master Touchscreen Task state is then set 416 and the to whether the previous zone has been marked 442. If a 
system returns to the top of the Polling Loop 500, previous zone has been marked, then the previous zone 

When a response to a zone map request occurs on marker is erased at step 444 and the new zone is marked 
queue 1 or 2 of the Polling Loop, the Security/Fire 446. In addition to marking the new zone, the Call ^nt 
Task (Task 1 or Task 2) jumps ( at 898 in FIG. 8c) to the 65 Zone Status subroutine 753. shown in FIG. 7/ is calM. 
security floor plan sub-menu subroutine, entering at If an affmnative determination is made at step 440 
point 424. Touches occurring while the system is wait- that a new zone has been touched, then step 446 is exe- 
ing for a response from the Security/Fire Interface wUl cuted. causing the marking of a new zone and the call- 
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ing of the Print Zone Status subroutine 753, The system 
then pauses for 0.1 seconds 448 and then determines 
whether more master touchscreen data is available 450. 
If an affirmative determination is made, then the subrou- 
tine returns to the top of the Polling Loop 500. If no 5 
more master touchscreen data is available, then a deter- 
mination is made as to whether a box has been touched 
on the current touchscreen 711a If no hox has been 
touched, then the subroutine returns to the top of the 
Polling Loop 500. 10 

If a box has been determined to have been touched at 
step 711, then, depending upon the function box 
touched, the security floor plan sub-menu subroutine 
will jump to the appropriate step to effectuate that func- 
tion 452. Thus, if the Quit box as shown in FIG. 3/ has 15 
been touched, the subprogram will return to the dor- 
mant screen 715. If the Back box has been touched, also 
as shown in FIG. 34 then the subroutine will remove 
the current "screen time out" ISIS event and insert new 
"screen time out" ISIS event 454. The subroutine will 20 
then initialize the security management sub-menu 456, 
which is shown in FIG. 3^ and display it on the touch- 
screen monitor. 

If the system arm/disarm function box has been se- 
lected at the box touched step 452, then the system 25 
jumps to the arm/disarm system substate or subroutine 
458 which is shown in FIG. 7j. If the zone enable/disa- 
ble function box was selected at step 452. then the sys- 
tem determines whether a zone has been selected at 460 
and, if a zone has been selected, jumps to the shunt zone 30 
substate at step 462. If no zone has been selected at step 
460, the subroutine returns to the top of the Polling 
Loop 500. If the highest alternate floor function box is 
selected at step 452, the subroutine sets the floor to the 
highest alternate floor at step 464 and then redisplays 35 
the selected security floor plan 466 (step 301 in FIG. 7A) 
on the touchscreen. If the lowest alternate floor func- 
tion box has been selected at 452, then the subroutine 
sets the floor to the lowest alternate floor 468 and then 
displays that security floor plan at step 466. (Step 301 in 40 
no. 7h) 

The Print Zone Status subroutine 753 is shown in 
flow chart form in FIG. 7/ The subroutine initially 
erases any previous zone status message 755 and then 
prints the selected zone's number, name and current 45 
status 757, A determination is then made as to whether 
the selected zone is shunted (that is, disabled) and 
whether the enable/disable touch box shown for exam- 
ple in FIG. 3/ displays the message "disable zone" 759. 
If the enable/disable touch box does display a "disable 50 
zone" message, then the "disable zone" touch box is 
replaced with the "enable zone" touch box 761. The 
subroutine then returns 763. 

If a negative determination is made at step 759, then 
a determination is made at step 765 of whether the se- 55 
lected zone is not shunted and the enable/disable touch 
box displays the "enable zone" message. If the determi- 
nation of that decision is negative, the program returns 
at 763. If, on the other hand, the determination of that 
decision is affirmative, then the "enable zone" touch 60 
box is replaced with the "disable zone" touch box 767 in 
a similar manner as described in connection with step 
761. The program then returns 763. 

If the arm/disarm system function box is selected at 
step 452, the security floor plan sub-menu subroutine 65 
branches to the arm/disarm system substate or subrou- 
tine 458 and first removes the current "screen time out" 
ISIS event and sets the touch flag to 0 at 470. A deter- 



mination is then made as to whether the security system 
is currently armed 472. If the security system is cur- 
rently armed, then the arm flag is reset at 474. If the 
security system is not currently armed, then the arm 
flag is set at 476. After the arm flag is either set or reset 
at steps 476 or 474, the voice synthesis system causes the 
speaker to transmit the phrase "Please enter your secu- 
rity pass code" to prompt the user to input the appropri- 
ate security system password so that the system will 
permit a change from the armed or disarmed state to the 
other. The user either speaks or in some other manner 
inputs the password and the subroutine then calls a Get 
Password subroutme 480 which captures the password. 
If no password is entered (a "null" password) 482, then 
the current security floor plan is redisplayed at 466. 

If a password is captured by the home automation 
system, then a determination is made as to whether the 
arm flag is set 484. If the arm flag is set, then an arm 
security system message and the password are sent to 
the Security/Fire Interface. The Security /Fire Task 
state is then set to the "arm system" state 488 (step 899 
in FIG. 8c). If it is determined that the arm flag is not set 
at step 484, then a disarm security system message and 
the captured password are sent to the Security/Fire 
Interface 490 and the Security/Fire Task state is then 
set to the "disarm system" state 492 (step 870 in FIG. 
8c). After the setting of the Security/Fire Task state to 
either the "arm system" or "disarm system" states at 488 
or 492, respectively, the subroutine sets the "Security 
Response Expected" ISIS event at 494 and informs the 
user to please wait by speaking that phrase 496 as well 
as displaying that message on the monitor 498. 

The system then sets the appropriate Master Touch- 
screen Task state 411 and returns to the lop of the Pol- 
ling Loop 500. The Master Touchscreen Task state is 
set such that the substate decoder will cause any 
touches on the touchscreen to display a cursor, but the 
selected state will iiot compare touches with the touch 
boxes on the menu. 

Once a result has been returned from the Security/- 
Fire Task or ISIS event (step 880 in FIG. Hb, step 895 in 
FIG. 8c step 1326 in FIGS. 13b and 13c). the system 
enters at point 491 and, depending upon the result 489, 
branches to one of six different places in the program. 
From point 489, the result can be determined to be 
good, which implies that the Security/Fire Interface 
has acknowledged the command, in which case the 
program then inserts a 'Vait for bad password" ISIS 
event 487, sets the subsute to prevent any user actions 
485 and then returns to the Polling Loop 500. If no "bad 
password" message is received from the Security/Fire 
Interface during the delay period, the ISIS event will 
return a result of "no bad password message during 
delay." The system then sends a zone map request to the 
Security/Fire Interface system interface, sets the 
Security/Fire Task state to "zone map request", sets a 
"security response expected" ISIS event 483, sets the 
appropriate substate 481 and then returns to the Polling 
Loop 500. 

If no response is the result determined at point 489. 
then the system speaks the phrase "no response" at ,479 
and then redisplays the current security floor plan 466. 
That would occur, if, for example, the message to the 
Security /Fire Inteiface was garbled in transmission. If, 
on the other hand> a **bad password" result is returned, 
the system is caused to speak the phrase "tmexpected 
response" at 475 and then redisplays the security floor 
plans at 466. 
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If the result at 489 is either "already armed" or "al- 
ready disarmed", then the system determines whether 
the arm flag has already been set 473. If the arm flag had 
been set» then the phrase "already armed" is spoken to 
inform the user of that slate at 471 and the current secu- 5 
rity floor plan is redisplayed 466. If the arm flag had not 
been set at 473, then the system speaks the phrase '*al- 
ready disarmed*' at 469 and then redisplays the current 
security floor plan 466. 

If the result at point 489 is the return of the zone map, 10 
then the system decodes the armed/disarmed status 467 
and detennines whether the system is armed or not at 

465. If the system is armed, then the phrase "system 
armed" is spoken 463 and the current security floor plan 

b redisplayed 466. If the system is not armed at 465, 15 
then the phrase "system disarmed** is spoken 461 and 
the current security floor plan is redisplayed at 466. 

If the shunt zone substate is selected at 462, as shown 
m FIG. li, then the program jumps to the shxmt zone 
subroutine shown at point 459 in FIG. IL The shunt 20 
zone subroutine then removes the "screen time out** 
ISIS event and says that sets the touch flag to 0 457. A 
determination is then made as to whether the security 
system is currently armed at 455. If the determination of 
that decision is affirmative, then the system speaks the 25 
phrase "Please enter your security pass code*' 453 so 
that the user may enter the appropriate password. The 
get password subroutine is called in order to capture the 
password entered by the user and then a determination 
is made as to whether a null password has been entered 50 
451. If a null password has been entered at point 451, 
then the appropriate security floor plan is redisplayed 

466, If the password entered is a potential password, 
then the shunt zone command and the password are sent 

to the security/fire interface 477 and the "security re- 35 
sponse expected*' ISIS event and the zone number for 
the ISIS event are stored 449. 

If, at step 455, a determination is made that the secu- 
rity system b unarmed, a shimt zone message is sent to 
the security/fire interface 447 and the ISIS event is set 40 
to "security response expected*' and the zone number 
for the ISIS event is stored 449. After storing the zone 
number for the ISIS event, whether the security system 
is anned or unarmed, the shunt zone subroutine then 
proceeds to set the Security/Fire Task to the ''shunt 45 
zone" state 445 and to inform the user to wait by both 
speaking the phrase "please wait for response form the 
security system" and by displaying "Please wait" as a 
message on the monitor screen 443. The appropriate 
Master Touchscreen Task substate is then set 441 to 50 
prevent further user actions until the security communi- 
cation is resolved, and the program then returns to the 
Polling Loop 500. 

The shunt zone substate is entered at point 439. De- 
pending upon the result determined at 437, the subrou- 55 
. tine then branches to one of several possible results. A 
first possible result is "good", which means that the 
Security/Fire Interface acknowledged the shunt zone 
command. A "wait for bad password" ISIS event is 
then inserted 487, the substate is then set 433 and the 60 
program returns to the PoUmg Loop 500. If the result at 
step 437 is that no response has been received, the sys- 
tem is caused to speak the phrase "no response" 479 and 
then redisplays the security floor plans. 

If the result at step 437 is a bad password, then the 65 
"w*ait for bad password" ISIS event is removed 431 and 
the phrase "invalid password" is spoken 429. The pro- 
gram then redisplays the security floor plans. If the 



result at step 437 is in an unexpected response, then the 
system speaks the phrase "unexpected response" 475 
and then redisplays the security floor plans. 

If the result at step 437 is that no bad password has 
been detected, the Security/Fire Task state is set to 
"zone map request** and a zone map request is sent to 
the Security/Fire Interface 427. The ISIS event is set to 
''security response expected" 435 and then the Master 
Touchscreen Task substate is set 425 to prevent further 
user actions. The program then rettoms to the polling 
loop 500. 

If the result at 437 is determined to be zone map 
returned, then a determination is made as to whether 
the current zone is now shimted 423. If the current zone 
is presently shunted, then the zone is shown as shunted 
on the touchscreen 421 and the system then decodes 
whether an armed or disarmed condition is present 419. 
If the current zone is not presently shunted, then the 
shunted indicator shown on the touchscreen is erased 
417 and the program moves to the code the arm/disarm 
condition at 419. 

The system then determines whether the security 
system is ready at 417. If the security system is ready, 
then a determination is made as to whether the security 
system was ready before the shunting occurred at 415. 
If the determination is made that the system was not 
ready before the shunting occurred, then the phrase 
"now ready" is spoken and the security floor plans 
subroutine continues to determine whether the security 
system is armed 411. 

If a determination is made at point 417 that the secu- 
rity system is not ready, then a determination is made as 
to whether the security system was not ready before the 
shunting occurred 409. If the outcome of that determi- 
nation is negative, then the phrase "now jiot ready" is 
spoken and the security floor plans subroutine then 
moves to a determination of whether the security sys- 
tem is armed at 411. In the event that the security sys- 
tem was not ready before shunting at 409 or whether 
the security system was ready before shunting at 415. 
the security floor plans subroutine then moves to deter- 
mine whether the security system is armed at 411. 

If the security system is armed at 411, then a determi- 
nation is made as to whether the security system was 
already armed before the shunting 405. If that determi- 
nation is negative, then the phrase "now disarmed" is 
spoken 403 and the system then determines whether the 
security system is armed 401. 

If the determination at step 411 is that the security 
system is not armed, then a determination is made as to 
whether the security system was disarmed before the 
shunting occurred at 300. If the outcome of that deter- 
mination is negative, the phrase "now disarmed" is 
spoken 302 and a determination is again made at 401 as 
to whether the security system is armed. If the outcome 
of the determination of whether the security system was 
disarmed before shunting at 300 is affirmative, the secu- 
rity floor plan subroutine then determines whether the 
security system is armed at 411. If the security system is 
anned at 411, then the current security floor plan is 
redisplayed 304. 

If the secxirity system is not armed at point 401. then 
the "please wait" message displayed on the screen is 
erased 306, the zone marker is erased and the current 
highlighted box on the screen is blanked at 308 and the 
system then displays the security system status at 310. 

The redormant or return to dormant screen routine of 
the Master Touchscreen Task module is shown in FIG. 
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7a From the redormant screen 312. the redormant sub- ty/Fire Interface data is available 826. If there is no 
routine turns ofT the voice response speakers 314 and more Security/Fire Interface data available, the pro- 
then determines whether an outgoing telephone call is gram returns to the top of the Polling Loop 500. 
active 316 If an outgoing telephone caU is active, then In the event that there is more Sccunty/Fire Inter- 
the system hangs up the phone 318. If no outgoing 5 face data available at step 826. or m the event that at the 
telephone call is active or after the system hangs up the V-state determination 802. the current state k V-state 3 
telephone at 318. a determination is then made as to a determination is made as to whether a valid command 
whether the video monitoring option is active at 320. If byte is present 828. If no valid cominand byte is present, 
the video monitoring function is active, then the redor- the V-state is reset 830. If a va^id command byte is 
mant subroutine turns ofT the video monitoring at 322. 10 present, a determination is xnade at step 832 as to 
After turning off the video monitoring at step 322 or if whether the length is greater than 1. If the outcorne of 
there is no active video monitoring, the redormant sub- that determination is affirmative, then the V^tate is set 
routine loads the dormant screen 324 and then deter- at step 834 to V-sute 4. If the detennmation of the 
mines whether a telephone call is presently incoming length at step 832 is such that the length is less Oian 1. 
326 If the incoming telephone line is active, then the 15 then the V-state at step 836 is set to V-state 5. After the 
message "System is currently responding to a telephone V-state is set to either V-state 4 or V^tate 5 at steps 834 
cair is printed and the program returns to the top of the or 836. respectively, a determinauon is agam made as to 
Polling Loop 500. If there is no incoming telephone call whether more Sccurity/Fire Interface data is avai ab e 
at step 326. ttien the Initrecognition subroutine is called 838. If no more Security/Fire Interface data is available 
330 md the program then returns to the top of the 20 at step 838. then the program returns to the top of the 
Polling Loop 500 Polling Loop 500. 

FIG Ip shows the Reset Synchronization routine 332 If more Security/Fu-e Interface data is available at 
of the Master Touchscreen Task module. After the step 838 or if the current V-state at step 802 is deter- 
Reset Synchronization routine 32 is called, that subrou- mined to be V-state 4. then the data byte is stored 840 
tine resets the Master Touchscreen V-state 334 and then 25 and a determination is made as towhether all data has 
restores the previous touch coordinates to the new been received from the Security/F^e Interface 842. If 
coordinates 336. The Reset Synchronization routine all data has been received from the Secunty/Fire Inter- 
then jumps to the master touchscreen immediate re- face, then the V-state is incremented by one at step 844 
sponse routine at point 338. allowing the immediate and a decision as to whether more Secunty/Fire Inter- 
rSponse routine to act upon the most recent touch 30 face data is pendmg is made at step 846. If it is deter- 
before the touch that got out of synchronization. mined at step 842 that all data has not been received. 

the Validation subroutine for the Security/Fire In- then the Validation subroutine moves d^ectly to step 
terfacc program is shown in FIGS. Sa and Sb, When it 846 to determine whether more Security/Fu-e Interface 
is determined that a new byte is in the Security/Fire data is available. If no more security/fire data is avail- 
Interface queue 800. the validation subroutine, depend- 35 able at step 846. then the program returns to the top of 
ine upon its current V-state 802. branches to one of the Polling Loop 500. 

several places in the Validation subroutine. If the V- If the answer to the query at st^ 846 is that more 
state is O; then a determination is made as to whether the Security/Fire Interface data is available, then the pro- 
byte is equal to the first synchronization byte 804. If it is gram goes to step 848, stores the nth data byte at rtep 
noL then the V-state is reset 806. If the determination at 40 840 and then goes to V-state 5. Obviously, as set forth m 
step 804 is affirmative, then the Validation subroutine FIG. 8^. if at step 802 the current state is V-state 5. then 
increments the V-state and inserts a •^complete security the program would branch durecUy to V-state 5 at 850. 
message" ISIS event 808. The subroutine then deter- As shown in FIG. 8^ from V-state 5 at step 850. the 
mines whether more Security/Fire Interface, data is program then removes the "wnnpieie secunty mess^^^^^^ 
avaUable 810 If there is no more Security/Fire Inter- 45 from the ISIS 852 and sets the V^itate back to 0 at 854. 
face data available, then the subroutine returns to the A determination is then made as to whether the check- 
top of the Polling Loop 500. sum byte is valid 856. If the checksum byte is not valid. 

If it is determined that more Security/Fire Interface then the program returns to the top of the Polhng Loop 
data is available at step 810 or if the current V-state is 500. If the checksum byte is determined to be valid at 
V-state 1. then a determination is made as to whether 50 step 856, then the program moves mto the unmediate 
the byte is equal to the second synchronization byte 812. response portion of the Security/Fire Interface subrou- 
If it is not, then the V-state is reset 814. If the current tine and acknowledges the message received to the 
byte is equal to the second synchronization byte, then Security/Fire Interface 858. A determmation is then 
the subroutine increments the V-sUte 816 and deter- made of the current state of the Secunty/Fire Interface, 
mines whether more Security/Fire Interface data is 55 whether it is in the Neutral State, the "disarm system 
available 818. If there is no more Security/Fire Inter- or other Contextual States, at step 860. If it is m the 
face data available then the program returns to the top Neutral State, the program proceeds to drtermme 
of the Polling Loop 500. whether the current command is an alarm at 862. All 

In the event that there is more Security/Fire Inter- commands that are not alarms are ignored and the pro- 
face dau available at step 818 or if the V-sUte upon 60 gram returns to the top of the Polling Loop 500. 
receiving a new byte in the Security/Fire Interface If the current command is detennmed to be an alann 
queue is V-state 2. as determined at step 802. then a at step 862 then, depending upon the type of alarm, the 
determination is made as to whether the incoming byte program wUI branch to one of three places. If the corn- 
is a valid length for a message 820. If it is not. then the mand at 862 is a fire alarm, then the program branches 
V-state is reset 822. If the new byte in the Security/Fire 65 to the Master Touchscreen Task fire alarm substate 864. 
Interface is of valid length, then the V-state is incre- If the command is determined to be a security alarm, 
mented by one and the length byte is stored 824. A then the program branches to the Master Touchscreen 
determination is next made as to whether more Securi- Task securit>' alarm substate 866. If the command at 
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step 862 is not a fire alarm nor a security alarm, then the 895. If the command at step 899 is determmed to be the 

program sets the lighting mood preset for . "all house password rejected command, then the result is set equal 

liehis" 868 and then jumps to the top of the Polling to rejected password (a rejected password means the 

^^^y^ 5QQ same as "bad password" in RG. Ik) and the program 

' If the state at step 860 is to determined to be the 5 goes to the Master Touchscreen Task arm/disarm secu- 

"disarm system" state, then the program determines rity system substate 895. (Step 491 in FIG. 7k) If the 

what type of command was received at step 870 and command at step 899 is determined to be any other 

branches to the appropriate point. If the command re- command than those previously discussed, then the 

ccived in the disarm system state is an acknowledgment. program executes to the Neutral State 878. 

then the program sets the result equal to good at step 10 FIGS. 13a thru ^/ show flow charts of the various 

872 and then jumps to the Master Touchscreen Task components of the ISIS events system. Referrmg first to 

disarm security system substate 880. (Step 491 at FIG. FIG. 13a, the screen time-out subroutine of the ISIS 

Ik) If the command at step 870 determines that the events system is shown. 

system has already been disarmed, then the result is set When a screen time-out ISIS event is called, the pro- 
to already disarmed at step 874 and the program then 15 gram jumps to the screen time-out ISIS event entry 
jumps to the master touchscreen disarm system substate point 1300. A determination is then made as to whether 
at step 880. If the command at step 870 is determined to the touch flag is equal to 1 at 1302. If the touch flag is 
be the rejection of the password inputted by the user, equal to 1, a new "screen time-out" ISIS event is m- 
then the result is set to equal password rejected at step serted at 1308, the touch flag is set to 0, and the subrou- 
876 and the program then moves to step 880. (Step 491 20 tine returns to the top of the Polling Loop 500. 
at FIG. Ik) If the command at step 870 is not one of the If a determination at step 1302 is made that the touch 
three commands already discussed, then the program flag is not equal to 1, then the Quit box on the touch- 
goes to the Neutral State at step 878. screen is blinked five times at point 1304. A determina- 
The other Contextual States in addition to the disarm tion is then made as to whether new touchscreen dau is 
system state are shown on FIG. 8c If, at step 860. the 25 avaUable at step 1306. If new touchscreen data is avaU- 
state is determined to be a shunt zone stote, then the able, the a new "screen time-out" ISIS event iS msened 
program determines which command within the shunt 1308 and the subroutine returns to the Polling Loop 
zone state has been received at step 882 and takes appro- 500. Inserting a new "screen time-out" ISIS event 
priate action. If an acknowledge command has been serves to maintain the present screen on the video mom- 
received at step 882. then the result is set to good at 884. 30 tor. 

If the command step at 882 is determined to be a re- If new touchscreen data is not available at step 1306. 
jected password, then the result is set to equal rejected then the subroutine returns to the dormant screen 1310. 
password at step 886. After the receipt of an acknowl- FIG, 136 is a flow chart showing the Security Re- 
edge or rejected password (a rejected password is the sponse Expected subroutine of the ISIS events system- 
same as a "bad password" in FIG. Ik) command and the 35 After jumping to the security response expected ISIS 
setting of the result at steps 884 or 886, the program event at 1312, the system determines whether this is the 
proceeds to the Master Touchscreen Task shunt zone first time a security response expected ISIS event is 
substate 890 (FIG. 7m, step 439), For all other com- being executed for that message at 1314. If it is the first 
mands received at step 882. the program executes the time for execution of that event, the message is sent to 
Neutral State 888. 40 the Security/Fire Interface again at step 1316 and a 
If the state at step 860 is determined to be the zone "security response expected" ISIS event is inserted at 
map request state, then a determuxation is made at step step 1318. The subroutine then returns to the Polling 
892 of whether the command received is an acknowK Loop 500. 

edge or rejected password or any other command. If If at step 1314 it is determined that the detected secu- 
thc command received is an acknowledge command. 45 rity response expected ISIS event is not being executed 
then the result is set to good at 894 and the program for the first time, then result is set to equal "no re- 
returns to the routine requesting the zone map 899. If a sponse" at 1320 and, depending upon the current master 
rejected password command is received as step 892, touchscreen state determination at step 1322, the sub- 
then the result is set to equal rejected password (a re- routine jumps to the zone map request entry point (step 
jected password is the same as "bad password" in FIG. 50 424 in FIG. 7g) at 1324. the arm/disarm security system 
7m) at step 896 and the program then returns to the substate entry point (step 491 in FIG. 7k) at 1326 or the 
routine requesting the zone m^ 898. For all other com- shunt zone substate entry point <step 439 in FIG. 7m) at 
mands received at step 892, the program returns to its 1328. 

Neutral State 878. FIG. 13c shows the flow chart for the Wait For Bad 

Lastly, if the Contextual State determmation at step 55 Password subroutine of the ISIS events. When a "wait 

860 indicates that the current state is the arm system for bad password" ISIS event is detected at 1330, the 

sute, then a determination is made at step 899 as to subroutine sets the result to "no bad password naessage" 

which command has been received. If an acknowledge during delay 1332 and then sets the Security/Fire Task 

command has been received at step 899, then the result state to Neutral 1334. A determination is then made of 

is set to good at step 897 and the program then goes to 60 the current master touchscreen state 1336. Depending 

the master touchscreen task arm/disarm security system upon the current state, the system cither then moves to 

substate 895. If the command received at step 899 is the the arm/disarm security system substate entry point 

already armed command, then the result is set to equal 1326 or to the shunt zone substate entry point 1328. 

already armed at step 893 and the program then goes to FIG. tSd shows a flow chart of the Complete Secu- 

the Master Touchscreen Task arm/disarm security sys- 65 rity Message subroutine of the ISIS event system. Upon 

tern substate 895. If the command at step 899 is deter- detecting a complete security message as an ISIS event 

mined to be not ready, then the result is set equal to not at 1338, the system resets the security/fire V-state 1340 

ready at step 891 and the program then proceeds to step and then returns to the top of the Polling Loop 500. 
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This is used to prevent incomplete messages from the security system entry point at FIG. 8c. steps 894 and 

Security/Fire Interface from disabling subsequent mes- 898. The zone map is returned prior to step 467 in FIG! 
sages. 

FIG. 14 is a flow chart showing the operation of a Depending upon the various combinations of current 
Secondary' Touchscreen Task module. The Secondary 5 and previous armed and ready conditions of the security 
Touchscreen Tasks are empty shells which serve only system, the system continues, and, by performing steps 
to pipe their data to the Master Touchscreen Task. 467, 465. 463 and 461, informs the user of any changes. 
Although not shown explicitly in the master touch- The follouong describes the Enable/Disable Security 
screen flow charts, the Master Touchscreen Task pulls Zone algorithm, assuming no errors occur, 
data from whatever queue corresponds to the current 10 Referring to FIG. 7a if the system is armed, at 455. 
active touchscreen. Thus, any secondary task may use the shunt zone command, zone number and password 
the master touchscreen routines by simply identifying are sent to the Security/Fire Interface at step 477. If. 
itself as the current active touchscreen and then jump- alternatively, the system is disarmed, then only the 
ing into the master touchscreen validation routines. The shunt zone command and zone nuinber is sent to the 
variable used to identify the current active touchscreen 15 Security/Fire Interface at step 447. The "Security Re- 
ts ACnVETSCRN. sponse Expected" ISIS event b set at step 449 and the 

At the Secondary Touchscreen Task, a determination Security/Fire Interface sends an acknowledgment, 

is first made as to whether a new byte is present in the The Security/Hre Task sets the result to good at step 

secondary touchscreen queue 1400. The secondary 884, in FIG. 8c That Task then jumps to the shunt zone 

touchscreen identifier is then stored to the variable 20 substate entry point from step 890. 

ACnVETSCRN 1402 and then the system jumps to Since the result is set to "good", a 'Vait for bad 

the Master Touchscreen Task validation routine 1404. password" ISIS event is inserted at step 487 in FIG. 7/n 

The following discussion describes the Arm/Disarm and then the master touchscreen substate is set to show 

Security System algorithm, assuming no errors occur. a cursor but not compare user touches with the display 

After the setting ofthe security/fire state to either the 25 boxes, at step 433. The purpose of the "wait for bad 

"arm system" or "disarm system" states at 488 or 492, password" ISIS event is to allow the Security/Fire 

respectively, the subroutine sets the "Security Response Interface time to respond with a "bad password" mes- 

Expected" ISIS event at 494. The Security/Fire Inter- sage in the event that the password sent is invalid, or the 

face then sends an acknowledgment at 858. security system was armed and no password was sent 

The Security/Fire Task sets the result to "good" 872 30 with the shunt zone command, 

and then jumps to the arm/disarm security system entry If the password sent was an invalid password, then 

point 872 m HG. $b or point 897 in FIG. 8c the Security/Fu-e Interface wiU send a "bad password" 

Since the result is now set to "good", a "wait for bad message. If the Security/Fire Task receives a bad pass- 
password" ISIS event is inserted as shown in FIG. 7k at word message, then it will set the result equal to "bad 
point 487. The master touchscreen substate is set to 35 password" at step 886 in FIG. 8f and jump to the shunt 
show a cursor at 485 but not to compare any touches zone substate entry point from step 890. 
with function boxes shown on the touchscreen. The If the result is "bad password", then the system 
purpose for the "wait for bad password" ISIS event is speaks "Security system reports an invalid password, 
to allow the Security/Fire Interface time to respond Please try again" at step 429 in FIG. 7m. 
with a "bad password" message in the event that the 40 Once the "wait for bad password" ISIS event times 
password sent is invalid. out. it will set the result equal to "no bad password 

In the event that the password sent was am invalid message during delay" at step 332 and then jump to the 

password, then the Security/Fire Interface will send a arm/disarm security entry point at step 1328. in FIG. 

"bad password" message, as shown on FIG. Sb at step 13c 

876 and FIG. 8c at step 889. If the Security/Fire Task 45 FIGS. ISa-lSd show the various routines associated 

receives a bad password message, then it will set the with the Input/Output Task module. ^ 

resuh equal to "bad password" and jump to the arm/- FIG. ISa shows a flow chart ofthe Validation routine 

disarm security system entry point at 876 in FIG. Sb or ofthe Input/Output Task. Upon determining that a new 

point 889 in FIG. 8c If the result is a "bad password", data byte is present in the input/output queue 1500. a 

then the system is caused to speak the phrase "Security 50 determination is made of the current V-state 1502 to 

system reports an invalid password. Please try again", which the program then jumps. If the current V-state at 

at step 475 in FIG. 7k. 1502 is V-state 0. then a determination is made as to 

Once the •'wait for bad password" ISIS event times whether the new data byte in the input/output queue is 

out at step 1330 m FIG. 13c. the result will be set to "no a valid command byte 1504. If the current byte is a valid 

bad password message during delay" at step 1330. the 55 command byte, the conmaand is stored 1506 and then a 

Security/Fu-e Interface state will be set to Neutral at determination is made as to whether the stored com- 

step 1334 and the system will then jump to the arm/dis- mand is an acknowledgment 1508. If the stored com- 

arm security system entry point 1326. mand is an acknowledgment, then the value of 1 is 

Since the result is "no bad password message during stored to the variable length 1510 and then the system 

delay", a zone map request is sent to the Security/Fire 60 jumps to the Immediate Response routines 1512. 

Interface at step 483 in FIG. 7k and the substate is set to If the current byte is not a valid command byte at step 

prevent touches from initiating any new action, as de- 1504, the system then returns to the top ofthe Polling 

scribed earlier in connection with FIG. 7k step 483. Loop 500. 

Those steps are performed in order to determine the If the command stored at step 1506 is not determined 

current armed or ready state of the security system. 65 at step 1508 to be an acknowledgment, then the value of 

The Security/Fire Interface then sends the requested 4 is stored as the variable length at step 1514 and the 

zone. The Security/Fire Task sets the result equal to "complete lO controller message" ISIS event is in- 

"zone map returned" and jumps to the arm/disarm serted 1516. A determination is then made as to whether 
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any more input/output data is available 1518. If no more Referring now to RG. X5d, a How chart of the On- 
input/output data is available, then the system returns To-Off and Off-To-On routines which may be called 
to the top of the Polling Loop 500. If more input/output from steps 1548 of FIG. ISb are shown. Smce both of 
data is available at step 1518 or if the V-state at step those routines operate in the same manner, only the 
1502 is determined to be V-state 1, then the new data 5 OfT-To-On routine 1548 has been Ulustrated. 
byte in the output queue is stored as data 1520 and then Once the Off-To-On routine has been called at 1548, 
a determination is made as to whether all data has been the program jumps to the current port number as deter- 
reccivcd 1522. If all data has not been received, then the mined at step 1562. Depending upon whether the cur- 
system returns to the top of the Polling Loop 500. If all rent port number is port N, port N-h 1 or port N-i-2. 
data has been received at step 1522, then the V-sUte is 10 etc.. the system then jumps to the appropnate port 
reset 1524 and the "complete I/O controller message" number and takes the appropriate action dependmg 
ISIS event is removed at step 1526. The system then upon the number of the bit in the bit mask and then 
jumps to the Immediate Response routines 1512. returns. 

FIG. ISb shows a flow chart of the Immediate Re- As will be apparent in light of the above, the Neutral 

sponse routine 1512 and also the Neutral State, both of 15 Sute of the input/output controller takes action based 

which are components of the Input/Output Controller upon the number of the input and type of transition. 

Task module. Upon jumping to the immediate response either ofT-io-on or on-to-off that caused an interrupt, 

routine, a determination is made at step 1528. of the For example, a message consisting of the numbers 101, 

present command. If the present command is an ac- 14, 255, 127 would be decoded as shown in Table 2: 

knowledgment, then the Immediate Response routine 20 TABLE 2 

does nothing and the system returns to the top of the ; 

Polling Loop 500 ' command indicating an intcmipung transition 

If the present COnmiand is an on-to-off or off-tO-on ,^ SSt« that the transition occurred on the I4th 

transition report, then the port number is decoded 1530 gf^op of 8 input modules, 

and the new bit mask and old bit mask are stored 1532. 25 255 - indicates the bit mask of a group of modules after 

The two bit masks are then compared to determine the transition (the new mask). . , . , _ 

I lie iwu uii iiu»f« « *ea4 127 - indicates the bit mask of a group of modules before 

which mputs caused the transiUon report 1534. ^ transition (old mask). 

If the command at step 1528 is a digital status com- 

mand, then the system decodes the port number, de- . . 1 u 1. 

codes the data bytes corresponding to the input board 30 As descnbed above, the two-bu masks would be 

1536 and then goes to determine the currem state at compared to determme that the 7th mput (readmg from 

1538 Likewise, after the new and old bit masks are 0 to 7) made an off-to-on transition and that the action 

compared and a determination as to which inputs assigned to that transition would be executed. Some 

caused the transition report is made at step 1534. a de- transitions may be ignored, smce they correspond to the 

termination is then made of the current state 1538. If the 35 return transition of a device. For example, a push-but- 

current state at step 1538 is the Neutral State, then a ton being pressed would cause an off-to-on transition 

determination is made as to whether the current com- report, but when it is released, it would cause an on-to- 

mand is a transition report 1540. If the current com- off transition report. The system controller 10 would 

mand is not a transition report, then all other commands take action on the off-to-on report, but not the on-to-off 

are ignored and the system returns to the top of the 40 report. 

Polling Loop 500. The following are some possible actions that would 
If the current command is a transition report, then for be taken based upon the detection of a transition: an- 
cach bit in the bit mask 1542. a determination is made as nounce the presence of a car in the driveway, announce 
to whether the transition was an off-to-on transition the presence of a person at the front door, announce 
1544. If it was not an off-to-on transition, then a deter- 45 that a bath is ready, and execute any action routines, for 
mination is made as to whether the transition was an example, in response to a transition that was caused by 
on-to-off transition 1546. If the determination at step a decorative touch switch. Such examples include light- 
1544 is that the transition was an off-to-on transition, ing moods, bath controls, vacation mode, random light- 
then the subroutine Off-To-On is called at 1548. In a ing, controlling voice alerts and setting a distributed 
similar manner, if the determination at step 1546 is that 50 audio system to send music to preset areas of the home, 
the transition was an on-to-off transition, then the sub- THE DEVICE DRIVERS 
routine On-To-Off is called at 1550. 

If the tranation at step 1546 is not an on-to-off transi- The Device Drivers perform the translation between 
tion. then a determination is made as to whether all bits the internal central processor commands and the corn- 
received were checked 1552. If not. the program checks 55 mands necessary to achieve the desired function. Since 
the next bit If all of the bits have been checked, then the there are many varieties of home subsystems, the cen- 
program returns to the Polling Loop 500. If the state at tral processor software has available a range of unique 
step 1538 is determined to be the read digital status device drivers to support each type of a certain type of 
state, then a determination is made as to whether the . subsystem. For example, although several different 
current command is a digital status command 1554. If 60 lighting control systems are known, and each has its 
the current command is not a digital status command, own unique device driver, all lighting mood device 
then the program enters the Neutral State 1556. If the drivers arc caUed in the same manner from within the 
current command is a digital status command, then the central processor software In the future, any new types 
"I/O controller response expected" ISIS event is re- of a subsystem could be easily incorporated into the 
moved 1558 and the program jumps to the read digital 65 present system by developing a new standard Device 
inputs Support Environment (step 1560) routine. The Driver for that type of the subsystem, 
read digital status Contextual Slate of the Input/Output Some of the more complex device drivers include a 
ControUer Task is shown in FIG. 15c. change to the current slate of the Task assigned to their 
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device. Alternatively, they may place an event on the 
Internal Scheduled Interrupt Server. For instance, 
some Device Drivers expect an acknowledgment from 
a secondary processor that their command has been 
understood and executed That acknowledgment will 
come back over a serial line and be processed by a Task, 
rather than by a Device Driver. In that case, the Device 
Driver will place that device's task in a state where it is 
expecting that acknowledgment. It would also place an 
event on the ISIS that would notify it if the acknowl- 
edgment had not been received within a certain period 
of time. 

The categories of Device Drivers available in the 
preferred embodiment are listed below in Table 3. 
However, the present home automation system is de- 
signed to allow for additional device drivers to be 
added as desired. 

TABLE 3 
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THE SUPPORT ENVIRONMENT 



The available Support Environment Functions are as 
shown in Table 5: 

TABLE 5 

Support Envifoninent Funciions: 
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Simple I>evice Driven are as follws: 

Analog inpoi driver 

Audio and video driver 

Bath and spa 

Bus digital input driver 

Digital output driver 

Lighting nKXxl driver without acknowledgment 

Random lighting driver 

Vacation mode driver 

Voice recognition 

VcMce synthesis (low priority) 

The following complcK device drivers are available: 

Lighting mood driver with acknowledgment 

Security Interface Driver 

Serial Digital Input Driver 

Voice Synthesis (high priority) 



The system utilizes the following action routines and 35 
Device Drivers as shown in Table 4: 

TABLE 4 



Display Curson 

PuIIbyie: 

Coksrchange: 

Instimeevent: 

Ddtiineevent: 
Checktouch: 

Reloadsched: 
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Action Routines & Device Drivers 



Musicon: 
MusicofT: 

HVACzone#setbackon: 

HVACzone^setbackofT: 

Unlockdoor. 

Lockdoon 

Bathtempveryhot: 

Bathtcmpbot: 

Bathtempwarm: 

Fdlthebath: 

Emptythebath: 

Sctvacmode: 

Resetvacmode: 

Raiuloinlightson: 

RandomUghtsofT: 

Ltinoods: 

Spcakerson: 

SpeakersofT: 

AMX: 

Semoioc: 

Scndtosec: 



Tom on distributed audio system to a 

set source to a set group of zones. 

Turn off distributed audio system. 

Sets a Heating/Cooling zone to an 

energy saving tcroperaTure. 

Sets a Heating/Cooling zone to a 

normal temperature. 

Unlocks a given door. 

Locks a given door 

Sets bath temperature to equivalent 

of ver>' hot 

Sets bath temperature to equivalent 
of hot. 

Sets bath temperature to equivalent 
of warm. 

Filb the bath to a preset level and 

temperature. 

Empties the bath. 

Turns on vacation mode for the home. 
Turns off vacation mode for the home. 
Turns on random lighting. 
Turns off random lighting. 
Sets a certain lighting mood sccordmg 
to the value passed it 
Turns on speech response speakers 
according to the vahies passed iL 
Turns off speech response speakers 
according to the values passed it. 
Fonnats and transmits a message to the 
Audio/Video Controller (IR Emitter). 
Fonnats and transmits a message to the 
lO ControUer. 

Formats and transmits a message to the 
S/F Interface. 
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Playafile: 

Pla>-memscquence: 



Loadscreen: 

Loadslice: 
Loadvocab: 

Initrecognition: 
Spchutil: 



Commbyport: 
Avrsofterror: 

Codeerron 
Readana: 

Readdigio: 

Metrabyteread: 



Metrabyteon 
MetrabyteofT: 
Metrapulseon: 
MctrapulseofT: 



Handles cursor movement and touchbox 
dynamics. 

Pulls a byte of data from a specified Serial 
Input Queue. 

Changes any color within a q>ecifled rec- 
tangular area to any other ct^or, according 
to the values passed it. 
Inserts an event into the ISIS queue, 
according to the event type and event time 
passed it. 

Deletes an event from the ISIS queue, 
according to the event time passed it. 
Compares recognition templates and touch 
coordinates with the locations of the current 
tottcbboxes on the screen, and returns the 
number of any activated touchbox. 
Reloads the daily schedule from a data fUe, 
screening out evenu that are not to occur 
on that day. 
Speak a phrase. 

Speak a phrase followed by individual 
words, such as numbers. i.e. The temp- 
erature is", **90r, "5", "degrees". 
Load a new menu screen and reset various 
graphics status variables. 
Load a portion of a screeiL 
Load a voice recognition vocabular>' into 
a speech board. 

Initialize voice recognition for a menu. 
A variety of routines designed to manage 
the speech boards in a cohesive, non-inter- 
fering maimer. 

A general serial transmission routine. 
Error handler for code^tccled error 
conditions. 

Error handler for software run-time errors. 
Reads an analog vcduge and/or temp- 
erature from an analog input- 
Reads the current values of all the inter- 
rupting digital inputs. 
Reads the current values of a group of 
passive digital inputs according to the value 
passed it. 

Turns on a digital output according to the 
values passed iu 

Turns off a distal output according to the 
values passed it. 

Pulses a digital output on for the duration 
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Pulses a digital output off for the duration 
passed it. 



There are several components of the support environ- 
ment, each of which wiD be discussed separately. They 
are: initialization, graphics routines, error logging and 
response, task switching and controls, and user- 
scheduled events. 

The initialization phase, as the name suggests, takes 
place when the home automation system is first run. At 
that time, various actions must be taken to ensure that 
the run-time portion of the software begins from a 
known state. Thus, all variables are initialized to values 
that reflect the most common expected state of the 
various subsystems. Interrupt and schedule queues are 
established to service the various devices. The devices 
on the AT bus 12 are reset and are brought to their 
active state. The remote secondary processors are sent 
interrogatory commands in order to ascertain that they 
are operative. 
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Additionally, any user-scheduled events subsequent 
to the current time and date are loaded into the ISIS. 
Depending upon the time of day, scheduled device 
driver routines may be activated, such as photo cell 
monitoring. At this point, the touchscreen "redormant" 5 
substate is executed to begin the run-time software. If 
any problems have occurred up to this point, the initial- 
ization software will report and log the problem and 
then attempt to recover and continue the initialization. 
Due to the modular nature of the system, and the fact 10 
that systems will be "rebooting" themselves subsequent 
to power failures, the software is designed to try to run 
a partially functioning system rather than to shut down 
completely. 

Another component of the support environment is 15 
the graphics routines. The EGA monitors placed 
throughout the home environment provide the major 
means of communicating information to the user. A 
variety of standard routines are thus provided in order 
to* control the usage of this screen. They consist of the 20 
following routines: load screen, load local image, high- 
light touch box and large font. 

The third component of the support environment is 
the error logging and response function. Within the 
system, there are two types of errors that may occur; 25 
Code errors and System errors. Code errors are those 
generated by the run-time portion of the BASIC com- 
piler. Those errors are of the type such as "divide by 0". 
Usually, the central processor software will recover 
from such errors, but, occasionally, the system may 30 
crash due to a particularly unpleasant error. Code er- 
rors are not reported to the user, but are logged with the 
error number, time, date and current state of the master 
touchscreen task. 

System errors, on the other hand, occur when a rou- 35 
tine detects a problem. These errors result from condi- 
tions that are actively examined by a particular software 
routine. Examples of system errors are voltage readings 
out of a normal range, failure of a secondary processor 
to respond, or inability to open a data file. System errors 40 
are logged with the error number, a text description of 
the error that may include some variable values, time, 
date, and the current state of the master touchscreen 
task. The central processor almost always recovers 
from system en'ors. System errors may or may not be 45 
reported to the user, at the option of the routine which 
detected the error. 

The fourth component of the support environment is 
the Task Switching and a Controls Subpart. As dis- 
cussed, all tasks are structured in approximately the 50 
same manner. Usually, states and state transitions are 
selected and executed based upon data processed from 
that task's assigned device or devices. However, a task's 
current state may occasionally be set or altered by an 
external software module, such as a device driver ex- 55 
pecting an acknowledgment over a serial line. ISIS 
events are also used to select and execute a state in a task 
under certain circumstances. Tasks may also select and 
execute a state within another task. Such external tasks- 
mandated state execution usually occurs between the 60 
master touchscreen task and another task. 

Due to the lack of multi-tasking capabilities in the 
operating system and the high-level language, there are 
no software controls to limit such nonstandard state 
selection and execution, as described above. Thus, the 65 
fimctionality of the nonstandard task-switching is 
checked through software validation and verification 
during software development. 
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The final component of the support environment is 
the User-Scheduled Events Module. The present home 
automation system allows the user to schedule events to 
occur at a future date and/or time, such as setting a 
"vacation mode" to occur while one is away for two 
weeks. The list of scheduled events is maintained in a 
disk file which is loaded when the program is run, and 
which is reloaded in the first few seconds of each day. 
When the schedule file is loaded, the dates of the events 
in the file are compared with the current date. If the 
dates coincide, then an event is inserted in the ISIS 
queue according to the time and type of event in the 
schedule file. That line' is then erased from the schedule 
file. 

OPERATION 

There are two phases to the operational sequence of 
the disclosed home automation system, the system ini- 
tialization and normal operation phases. When the home 
automation system is fully installed in a home and the 
power switch is then turned on, the system initialization 
phase begins under the control of a batch file of operat- 
ing system commands. Near the end of the initialization 
phase, the home automation real-time software, 
HEART is loaded and control is then passed to that 
system. The HEART software performs the remaining 
initialization tasks and then enters the normal operation 
phase. Each of the system initialization and normal 
operation phases is described below in more detail. 

At power on, the system initialization phase begins. 
The central processor 10 executes its standard boot-up 
procedure and runs a predetermined auto-execute batch 
file designed to initialize the disclosed home automation 
system. In the preferred embodiment, this batch file is 
set up as a series of MS DOS commands. The main 
functions of this batch file are to configure the present 
home automation system hardware and to load data 
files into memory. The initial portion of the batch files 
serves to change hard disk directories where data files 
are located and copies files from those directories to the 
system extended memory or RAM. That is done so that 
the computer can load and use the data more rapidly 
than if it had to rely on hard disk access. The following 
are examples of the various classes of files which are 
stored as shown in Table 6: 

TABLE 6 

*.pic: Full screen color graphics files in a Tonnat 

compatible with the hardware's color graphics 
controller board. These files are loaded by the 
HEART program into graphics memory to show a 
ftill-screen. high-resolution, color-graphics 
display on a monitor or touchscreen. 

*.{nt: Color graphics type fonts consisting of large 

characters and symbols to be dynamically 
di^Iayed <hi the scr ee n. 

*.icn: Icons and other graphics images deagned to be 

di^layed on the screen under program control. 

*.sct: Multi-word or complete phrase Speech Response 

data files in the proper format for the speech 
response hardware. 

*.sfk: Sizigle word or partial phrase Speech Response 

datt files in a format for appending with other 
.SPK files to build variable speech phrases 
compatible with the TISPEECH board. 

*.voc: Vocabulary template data compatible with the 

TISPEECH board*s recognition mode. When these 
files are loaded into the TISPEECH board by the 
HEART program, they define the active 
recognition vocabular>'. 

♦.img: software files containing routines to be loaded 

for the TISPEECH board. Depending on which 
routine is loaded, the TISPEECH board will 
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main menu touchscreen task operates as a general Con- 
TABLE 6K:ontmued ^^^^^ 5^^^^^ ^ ^^^^^ ^ ^o^, ^^^^ form in FIG. M 

pcrfonn voice recogniiion, ^>eecb response, or Unlike the dormant State, however, where, in this exam- 

, tekphone functions. ; . pj^^ Q^Iy one action was possible through the touch- 

5 screen, the main menu slate will perform different func- 

After all the files have been copied, the auto-execute ^j^j^j depending upon where the touchscreen is 

batch file runs the other initialization software neces- touched. Once again, a display cursor "follows" the 

sary to initialize system hardware. One such example of user's finger as he slides it about the screen, but now, 

such initialization software is a program called tispeech ^ijgn his finger moves across any touch active area or 

which is run to initialize the TISPEECH hardware jq f^^tion box (called a box in FIG. that area is high- 

contamed in the speech processor 58. lichted by changing its color. If the user lifts his finger 

The last function of the auto-execute batch file is to an area is highlighted, then the software wUl 

load and run the HEART operating software. recognize an input and perform the action appropriate 

me HEART software begins by performing i« tKcation in the ^ menu state. (See bottom of 
initialization routines. Variables are mitialized, the mter- jj 

Kn. the HEART software executes the touchscreen agement menu, as shown m FIG 3g Tlie software then 

"^X^Te^m^^^^^^ is a menu scrL of th\available security feature. Gt 

initialized, the normal operation phase begins. In nor- should also be noted that the Secunty M^gement 

mal operation, the system usually appears to be "dor- state is also a Contextual Slate as shown in FIG. Id) 

manr* with no apparent actions being performed. How- In this example, the user will disable a smgle secunty 

ever, the system software is busy in its PoUing Loop zone so that a window can be left open, and then arm 

(shown in FIGS. 5a-Sc) checking all of the input queues the security system using a floor plan screen display. To 

for data. Based upon the data received at the mput do this, the user touches the menu box labelled ENTRY 

queues, the present home automation system will take LEVEL on FIG. 3g. smce the zone to be disabled is on 

appropriate action. An example of one of the possible that floor. The menu box is highlighted, and when the 

operational sequences follows. user lifts his finger, the display changes to show a floor 

In order to describe the operation of the disclosed p]^^ of the entry level of the home, FIG. 3k 

home automation system in normal use, the following After displaying the floor plan, the home automation 

describes how the user would utilize the touchscreen to system requests the status of all of the security zones on 

control the security system. It should be understood, ^'^^ j^y^j f^om the security interface 14, as shown in 

however, that controDing the security system is only pjQ 7^ ^^^j^^ system displays the security sutus of 

one of the functions as described herein, but is beUeved ^^j, ^Q^e on the display, as shown in FIG. 7A. This 

to be representative of how the instant home automa- display, which is shown in FIGS. 3h and 3i\ indicates 

tion system operates. The foUov^dng example shows. ^^jj 3 colored icon whether each zone is enabled, dis- 

how, using touchscreen commands, a user would dis- ^^j^ ^^^^ touchscreen state is set to analyze 

able a security zone, and then arm his security system. ^^^^ touches in the middle of FIG. 7A). 

In preferred installation, the user steps up to the ^^^^ disable the right living room window, the 

touchscreen which is recessed in a wall and is display- selects that window by touching it on the floor 

ing a "dormant screen" display, which can range from display. The system responds by highlighting the 

a blank screen to a logo to a custom graphic designed ^dow with a colored box, and describing the associ- 

for the user. At this point, the Master Touchscreen Task ^ security zone and its status at the bottom of the 

is m the Neutral State and the software is in the Polling disable the zone, the user then touches the 

Loop, FIGS. Sa-Sc To enter a command, the t^er ^^^^ l^heUed disable zone. (See FIG. 7i). The 

touches the display screen anywhere on its surface. Tlie ^ responds by requesting the home security sys- 

user's touch causes the touchscreen mtemal electromcs ^ ^^^j^ ^^^^ confirmation of its 

to begin sending touch position locations on a conmiu- ^ ^^^^ ^^^^ ^^^^^ 

nications cable connected to a serial port sy^jf jf ^en the zone disable request is accomplished and 

controller, mmiely through ser^interface^^^^ confirmed, the system modifies the zone icon and 
dation routine shown m FIG. 7a »«^^fTPf ^^^^^^^ 55 changes tl^e sutus description on the screen, as shown 

enters the Immediate Response routme shown m FIG. 55 ^ ^g^ accomplishing those 

J xkTUs^ tiw. «CM- lift* hK fineer from 60 box labelled ARM SYSTEM. (See FIG. 7i). lie oox is 

S:*^'^ f^ng^S^^O.^^ds'^^^^^ Wghlighted and *e dlsplay^changes to a pass «xJe 

tome a^mation will recognize that as a valid display as shown m FIG 3h. (The n«>w chart of R^^^ 

S ™cnt on the dormant screeTspeak an optional calls another Contextual State, which .s shown m F\G. 

ereeting and then initialize the main menu state, which ^d). 

^shov.^ as a flow chart in FIG. 7c 65 The n«t step ^ for the user to Mter a secum^^ 

WhDe in the main menu mode, the system displays a word code, one digit at a time, followed by the com^ 

new graphic such as that shown in FIG. 3d. a menu of mand OK. If the code is correct, the system sends an 

the general features that the system can perform. The "arm" request to the secunty system, as shown m FIG. 



55 



5,086,385 



56 



3/ When confirmation is received, the Entry Level 
screen is again displayed with the green "ready to arm" 
indicator replaced by a red "armed" indicator as shown 
m FIG. 3/n. 

The last step in this example is for the user to conrniu- 5 
nicate to the home.automation system that he has com- 
pleted his task and is through with the system. To do 
this, he touches the white box marked ''Quit" which 
highlights and returns the system to its dormant screen 
and dormant state. Thus, from the flow chart shown in 10 
FIG. 7i the system jumps to the redormant routine flow 
chart, which is shown in FIG. 7a 

As has been described previously herein, the present 
inventive expandable home automation system can be 
utilized to control numerous environments. One such 15 
environment which can be controlled in a mamier simi- 
lar to that disclosed herein is a media or conference 
room, as shown in FIGS. X2a~12g, 

As shown in FIG. 12a» the main menu of a touch- 
screen control-based system similar to that disclosed 20 
herein in connection with the expandable home automa- 
tion system is shown. Equipment such as audio equip- 
ment, TV tuners. VCRs, lights and drapes, overhead 
projection and slide presentation equipment can be con- 
trolled merely by touching the appropriate function box 25 
on the touchscreen menu shown in FIG. 12a, Other 
functions such as quitting, muting the sound, volume up 
and down, and lights on and off may also be provided. 

FIG. 12b shows the audio sub-menu touchscreen for 
the main menu touchscreen shown in FIG. 12a. In addi- 30 
tion to containing the functional control block con- 
tained m the main menu touchscreen of FIG. 12c, the 
audio sub-menu touchscreen allows the user to enable a 
plurality of high fidelity speakers, four being shown in 
this specific example. 35 

FIG. 12c shows the TV tuner sub-menu touchscreen 
which is selected by touching the TV tuner functional 
block of FIG. 12a, The TV tuner sub-menu touchscreen 
allows the user to turn the TV on by selecting a particu- 
lar channel, to turn the TV off, and to scan the channels 40 
using up and down fimction blocks. 

FIG. I2d shows the VCR sub-menu touchscreen 
which is displayed upon selecting the VCR function 
block of FIG. 12fl. The VCR can be actuated and 
placed in the play, pause, fast-forward, rewind or stop 45 
mode, it can be turned off and an indication is provided 
as to whether the video tape cassette has been loaded 
into the VCR. 

FIG. 12^ shows the lights and drapes sub-menu 
touchscreen which appears upon selecting the lights 50 
and drapes function from the main menu touchscreen 
shown in FIG. 12fl. Various predetermined settings can 
be chosen by selecting* scenes 1^, which actuate the 
lights and drapes according to predetermined data in a 
manner similar to that discussed in connection with the 55 
lighting moods sub-menu touchscreen FIG. 3^. In addi- 
tion, functional blocks are provided to actuate the 
drapes to change their state either from open to close or 
closed to open and to turn the lights off. 

FIG. 12/ shows the overhead projection sub-menu 60 
touchscreen which appears on the monitor upon select- 
ing the overhead projection functional box from the 
main menu touchscreen shown in FIG. 12fl. The touch- 
screen menu of FIG. 12/ allows the user to turn on the 
overhead projector, turn the lighte off and to move the 65 
projection screen up and down. 

FIG. 12g shows a slide presentation sub-menu touch- 
screen which is selected upon selecting the slide presen- 



tation function from the main menu touchscreen shown 
in FIG. 12a. A plurality of projectors may be actuated 
and the lights turned on and off using the functional 
blocks provided on this sub-menu touchscreen. In addi- 
tion, the selected projector can be focused in and out 
and caused to move in either a forward or reverse direc- 
tion, using additional functional touch blocks provided 
on this sub-menu touchscreen. 

Although many of the functions described herein 
operate to set parameters to those values preset in data 
files, it will of course be obvious to those of ordinary 
skill in the art that the present expandable home auto- 
mation system can also be utilized to generate ramp 
signals with which to continuously vary the settings to 
a contemporaneous user determined value. 

Although only a preferred embodiment is specifically 
illustrated and described herein, it wQl be appreciated 
that many modifications and variations of the present 
invention are possible in light of the above teachings 
and within the purview of the appended claims without 
departing from the spirit and intended scope of the 
invention. 

What is claimed is: 

1. A system for automatically controlling a plurality 
of remotely controllable subsystems within a house, said 
system also being useful for automatically performing a 
plurality of commands input to the system by a user, 
said system comprising: 

a programmed data processor, 

at least one data interface means connected to said 
programmed data processor by means of a data 
bus, said at least one data interface means being 
connected to said plurality of remotely conuolla- 
ble subsystems for providing bidirectional direct 
communication between the subsystems and said 
programmed data processor, 

visual communication interface means comprised of a 
high resolution video monitor and associated 
touchscreen interface, through which the system 
communicates information received from said sub- 
systems and also accepts commands from said user, 
said commands being communicated to said system 
merely by the user touching said touchscreen inter- 
face means; and 

audio communication interface means comprised of a 
voice recognition and sf>eech system, through 
which the system and user conmiunicate with each 
other, said voice recognition and speech system 
functioning together with said visual communica- 
tion interface to provide cues to said user as to 
available commands, feedback of whether the cur- 
rent conmiand has been accepted by the system and 
the results of performing said current conunand. 

2. The system of claim 1, further including a modem 
through which said system can communicate with ex- 
ternal information retrieval data bases. . 

3. The system of claim 1, further including a plurality 
of types of user devices simultaneously coimectcd to 
said programmed data processor for communicating 
commands to said system, said user devices comprising 
at least one of keypads, serial data keypads, hand-held 
remote controls, computer keyboards and telephones. 

4. The system of claim 1, wherein said plurality of 
data interface means includes translator means, con* 
nected between a commercially available automation 
controller and said progranmned data processor, 
whereby said system can control said automation con- 
troller. 
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5. The system of claim 1. wherein said visual commu- data keypad, a voice recognitibn system, hand-held 
nicaiion Lntcrface means provides the user with a floor remote control, computer keyboard and telephone, 
plan display of at least one floor of said house, by which said at least two additional communication inter- 
the user may enter certain commands into the system by face means being simultaneously connected with 
touching the appropriate portion of said floor plan. 5 each other and said visual conrmiunication interface 

6. The system of claim 1. further including home means to said programmed data processor such 
appliances connected to said data interface means such that said user may communicate commands to said 
that said system can control said home appliances. system using any of said connected communication 

7. The system of claim 1, wherein the user may sched- interfaces. 

ule the occurrence of user^etennined events by the 10 12. The system of claim 11, wherein said system is 

system by utilizing said visual communication interface expandable to control a plurality of remotely controUar 

nieans. We subsystems and home appliances by incorporating 

8. The system of claim 1, wherein said remotely con- an unlimited number of data interface means, 
trollable subsystems comprise at least one of home heat- 13. The system of claim 11, further including a 
ing and cooling systems, access control systems, secu- 15 modem through which said system can commumcate 
rity systems and lighting systems. with external information retrieval data bases. 

9. The system of claim 1, wherein one of said plurality 14. The system of claim 11. wherem said plurahty of 
ofremotely controllable subsystems comprises audio/v- data interface means includes translator means, con- 
ideo entertainment means which may be controlled by nectcd between a commercially available automation 
the user by operation of said visual communication 20 controller and said programmed data processor, 
interface means. whereby said system can control said automation con- 

10. The system of claim 1, wherein said system is troller. 

expandable to control a plurality ofremotely controUa- 15. The system of claim 11. wherein said visual com- 

ble subsystems and home appliances by incorporating munication interface means provides the user with a 

an unlimited number of data interface means. 25 floor plan display of at least one floor of said house, by 

11. A system for automatically controlling a plurality which the user may enter certain commands into the 
of remotely controllable subsystems within a house, said system by touching the appropriate portion of said floor 
system also being useful for automatically performing a plan. 

plurality of commands input to the system by a user, 16. The system of claim 11, further including home 
said system comprising: 30 appliances connected to said data interface means such 
a programmed data processor; that said system can control said home appliances, 
a plurahty of data interface means connected to said 17. The system of claim 11, wherein the user may 
programmed data processor by means of a data schedule the occurrence of user-determined events by 
bus. said plurality of data mterface means being the system by utilizing said visual communication inter- 
connected to said plurality of remotely controUa- 35 face means. 

ble subsystems for providing bidirectional direct 18. The system of claim 11, wherein the user may 

communication between the subsystems and said schedule the occurrence of user-determined events by 

programmed data processor; the system by utilizing said voice recognition system. 

visual communication interface means comprised of a 19. The system of claim 11. wherein said remotely 
high resolution video monitor and associated 40 controllable subsystems comprise at least one of home 
touchscreen interface, through which the system heating and cooling systems, access control systems, 
communicates information received from said sub- security systems and lighting systems, 
systems and also accepts commands from said user. 20. The system of claim 11, wherein one of said plu- 
said commands being communicated to said system rality of remotely controllable subsystems compnses 
merely by the user touching said touchscreen inter- 45 audio/video entertainment means which may be con- 
face means; and trolled by the user by operation of said visual communi- 

a nlurality of additional communication interface cation interface means. 

^ 1 * * r- 1,* «J » • • ♦ ♦ 



means, including at least two of a keypad, serial 
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